[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:15 dist-test-slave-tfm9 krb5kdc[19678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:15 dist-test-slave-tfm9 krb5kdc[19678](info): set up 2 sockets
Sep 24 13:42:15 dist-test-slave-tfm9 krb5kdc[19678](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 13:42:16.459149 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:16 dist-test-slave-tfm9 krb5kdc[19678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:42:16.576236 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:43229
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:43229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:16.655932 19691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:16.656142 19691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:16.656165 19691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:16.657310 19691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:16.657351 19691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:16.657366 19691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:16.657378 19691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:16.657428 19691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:16.658946 19691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:43229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:43229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:16.659200 19691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:16.659421 19691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:42:16.664561 19691 server_base.cc:1034] running on GCE node
W20260924 13:42:16.664608 19697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:16.664608 19696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:16.664551 19699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:16.665122 19691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:16.665419 19691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:16.666594 19691 hybrid_clock.cc:648] HybridClock initialized: now 1790257336666577 us; error 45 us; skew 500 ppm
Sep 24 13:42:16 dist-test-slave-tfm9 krb5kdc[19678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257336, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:16.669026 19691 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:42:16.669395 19691 webserver.cc:460] Webserver started at http://127.19.53.254:32901/ using document root <none> and password file <none>
I20260924 13:42:16.669579 19691 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:16.669620 19691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:16.669705 19691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:16.670742 19691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "528f9b4f4a484bc593bcd873bf709b6d"
format_stamp: "Formatted at 2026-09-24 13:42:16 on dist-test-slave-tfm9"
server_key: "15463bb8fbba9ffd91d83f11e5f7db78"
server_key_iv: "aa9e85e979aa79bc1a26b1f23b1ed85c"
server_key_version: "encryptionkey@0"
I20260924 13:42:16.671054 19691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "528f9b4f4a484bc593bcd873bf709b6d"
format_stamp: "Formatted at 2026-09-24 13:42:16 on dist-test-slave-tfm9"
server_key: "15463bb8fbba9ffd91d83f11e5f7db78"
server_key_iv: "aa9e85e979aa79bc1a26b1f23b1ed85c"
server_key_version: "encryptionkey@0"
I20260924 13:42:16.672326 19691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:42:16.672997 19706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:16.673215 19691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:16.673300 19691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "528f9b4f4a484bc593bcd873bf709b6d"
format_stamp: "Formatted at 2026-09-24 13:42:16 on dist-test-slave-tfm9"
server_key: "15463bb8fbba9ffd91d83f11e5f7db78"
server_key_iv: "aa9e85e979aa79bc1a26b1f23b1ed85c"
server_key_version: "encryptionkey@0"
I20260924 13:42:16.673350 19691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:16.687922 19691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:16.690112 19691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:16.690212 19691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:16.693812 19691 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:43229
I20260924 13:42:16.693866 19758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:43229 every 8 connection(s)
I20260924 13:42:16.694216 19691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:16.695106 19759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:16.697232 19759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d: Bootstrap starting.
I20260924 13:42:16.697803 19759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:16.698107 19759 log.cc:824] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:16.698799 19759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d: No bootstrap required, opened a new log
I20260924 13:42:16.700301 19759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528f9b4f4a484bc593bcd873bf709b6d" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43229 } }
I20260924 13:42:16.700433 19759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:16.700454 19759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528f9b4f4a484bc593bcd873bf709b6d, State: Initialized, Role: FOLLOWER
I20260924 13:42:16.700628 19759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528f9b4f4a484bc593bcd873bf709b6d" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43229 } }
I20260924 13:42:16.700711 19759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:16.700754 19759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:16.700835 19759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:16.700902 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 19691
I20260924 13:42:16.701037 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:16.701439 19759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528f9b4f4a484bc593bcd873bf709b6d" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43229 } }
I20260924 13:42:16.701563 19759 leader_election.cc:302] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 528f9b4f4a484bc593bcd873bf709b6d; no voters:
I20260924 13:42:16.701735 19759 leader_election.cc:288] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:16.701732 19671 external_mini_cluster.cc:1428] Setting key 3f6c1192d190b5d7bbf2153bcfddf152
I20260924 13:42:16.701771 19762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:16.701900 19762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [term 1 LEADER]: Becoming Leader. State: Replica: 528f9b4f4a484bc593bcd873bf709b6d, State: Running, Role: LEADER
I20260924 13:42:16.702014 19759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:16.702077 19762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528f9b4f4a484bc593bcd873bf709b6d" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43229 } }
I20260924 13:42:16.703120 19763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 528f9b4f4a484bc593bcd873bf709b6d. Latest consensus state: current_term: 1 leader_uuid: "528f9b4f4a484bc593bcd873bf709b6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528f9b4f4a484bc593bcd873bf709b6d" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43229 } } }
I20260924 13:42:16.703207 19763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:16.703399 19777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:16.703616 19764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "528f9b4f4a484bc593bcd873bf709b6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528f9b4f4a484bc593bcd873bf709b6d" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43229 } } }
I20260924 13:42:16.703751 19764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:16.704030 19777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:16.705646 19777 catalog_manager.cc:1347] Generated new cluster ID: 657941399ec54cce94f3e415eedc7f53
I20260924 13:42:16.705698 19777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:16.713182 19777 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:16.713871 19777 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 13:42:16 dist-test-slave-tfm9 krb5kdc[19678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:16.718761 19777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 528f9b4f4a484bc593bcd873bf709b6d: Generated new TSK 0
I20260924 13:42:16.718917 19777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:42:16.720077 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:16.702659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.19.53.254:43229)
0924 13:42:16.703231 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:16.703241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:16.703878 (+ 637us) server_negotiation.cc:407] Connection header received
0924 13:42:16.706170 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:16.706177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:16.706231 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:16.706374 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:16.708186 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:16.709671 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:16.711119 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:16.711759 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:16.711871 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:16.716173 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:16.716177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:16.716177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:16.716188 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:16.718102 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:16.718609 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:16.718612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:16.718612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:16.718715 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:16.719030 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:16.719032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:16.719032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:16.719260 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:16.719359 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:16.719730 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 13:42:16.719838 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":456,"threads_started":1}
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:42:16.744881 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:43229
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:16.824934 19785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:16.825132 19785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:16.825155 19785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:16.826341 19785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:16.826382 19785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:16.826408 19785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:16.827822 19785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:43229
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:16.828047 19785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:16.828294 19785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:16.833366 19790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:16.833380 19791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:16.833384 19793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:16.833540 19785 server_base.cc:1034] running on GCE node
I20260924 13:42:16.833767 19785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:16.834028 19785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:16.835173 19785 hybrid_clock.cc:648] HybridClock initialized: now 1790257336835143 us; error 46 us; skew 500 ppm
Sep 24 13:42:16 dist-test-slave-tfm9 krb5kdc[19678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257336, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:16.837522 19785 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:42:16.837963 19785 webserver.cc:460] Webserver started at http://127.19.53.193:44219/ using document root <none> and password file <none>
I20260924 13:42:16.838231 19785 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:16.838299 19785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:16.838426 19785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:16.839421 19785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "2946c2c4b0a241c1af179100e4942685"
format_stamp: "Formatted at 2026-09-24 13:42:16 on dist-test-slave-tfm9"
server_key: "cbb0ce0c50b2a62abe2813e984d24227"
server_key_iv: "e558d1e2f80511b7fcc1e7aa48a94bee"
server_key_version: "encryptionkey@0"
I20260924 13:42:16.839761 19785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "2946c2c4b0a241c1af179100e4942685"
format_stamp: "Formatted at 2026-09-24 13:42:16 on dist-test-slave-tfm9"
server_key: "cbb0ce0c50b2a62abe2813e984d24227"
server_key_iv: "e558d1e2f80511b7fcc1e7aa48a94bee"
server_key_version: "encryptionkey@0"
I20260924 13:42:16.840979 19785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:16.841780 19800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:16.841987 19785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:16.842092 19785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "2946c2c4b0a241c1af179100e4942685"
format_stamp: "Formatted at 2026-09-24 13:42:16 on dist-test-slave-tfm9"
server_key: "cbb0ce0c50b2a62abe2813e984d24227"
server_key_iv: "e558d1e2f80511b7fcc1e7aa48a94bee"
server_key_version: "encryptionkey@0"
I20260924 13:42:16.842176 19785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:16.857609 19785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:16.859462 19785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:16.859625 19785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:16.859838 19785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:16.860148 19785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:16.860203 19785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:16.860245 19785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:16.860273 19785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:16.865979 19785 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:35571
I20260924 13:42:16.866029 19913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:35571 every 8 connection(s)
I20260924 13:42:16.866366 19785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:42:16.869212 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 19785
I20260924 13:42:16.869311 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:42:16.869457 19671 external_mini_cluster.cc:1428] Setting key e19ae4267a988c00940239c3aef8680d
Sep 24 13:42:16 dist-test-slave-tfm9 krb5kdc[19678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257336, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:16.876761 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:16.867071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:50363 (local address 127.19.53.254:43229)
0924 13:42:16.867115 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:16.867117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:16.867337 (+ 220us) server_negotiation.cc:407] Connection header received
0924 13:42:16.867883 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:16.867885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:16.867898 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:16.867946 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:16.868922 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:16.869491 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:16.870739 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:16.871074 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:16.871147 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:16.873609 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:16.873611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:16.873611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:16.873617 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:16.875128 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:16.875635 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:16.875636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:16.875636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:16.875691 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:16.876096 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:16.876097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:16.876097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:16.876240 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:16.876296 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:16.876629 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 13:42:16.876699 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:16.877018 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:16.867025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:43229 (local address 127.19.53.193:50363)
0924 13:42:16.867219 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:16.867253 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:16.867756 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:16.868013 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:16.868017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:16.868617 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:16.868813 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:16.868817 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:16.869618 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:16.869619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:16.870647 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:16.870649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:16.871153 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:16.871154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:16.871250 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:16.871805 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:16.871816 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:16.873512 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:16.875222 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:16.875229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:16.875232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:16.875549 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:16.875787 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:16.875788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:16.875788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:16.875927 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:16.876309 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:16.876315 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:16.876483 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 13:42:16.876606 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 13:42:16.876775 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260924 13:42:16.877507 19914 heartbeater.cc:347] Connected to a master server at 127.19.53.254:43229
I20260924 13:42:16.877622 19914 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:16.877789 19914 heartbeater.cc:511] Master 127.19.53.254:43229 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
I20260924 13:42:16.878325 19723 ts_manager.cc:194] Registered new tserver with Master: 2946c2c4b0a241c1af179100e4942685 (127.19.53.193:35571)
I20260924 13:42:16.878957 19723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:50363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:42:16.898283 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:43229
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:16.985054 19919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:16.985247 19919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:16.985270 19919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:16.986539 19919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:16.986582 19919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:16.986609 19919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:16.988004 19919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:43229
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:16.988268 19919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:16.988492 19919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:42:16.993536 19919 server_base.cc:1034] running on GCE node
W20260924 13:42:16.993489 19927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:16.993503 19925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:16.993503 19924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:16.993965 19919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:16.994221 19919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:16.995358 19919 hybrid_clock.cc:648] HybridClock initialized: now 1790257336995346 us; error 29 us; skew 500 ppm
Sep 24 13:42:16 dist-test-slave-tfm9 krb5kdc[19678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257336, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:16.997781 19919 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:42:16.998255 19919 webserver.cc:460] Webserver started at http://127.19.53.194:46111/ using document root <none> and password file <none>
I20260924 13:42:16.998479 19919 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:16.998548 19919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:16.998715 19919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:16.999711 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "83406e20fcb14c6ba675854f1e61b401"
format_stamp: "Formatted at 2026-09-24 13:42:16 on dist-test-slave-tfm9"
server_key: "182e87736c12707632a384946731ef41"
server_key_iv: "7dbd5ed117d47bcf5f0db65e368d777e"
server_key_version: "encryptionkey@0"
I20260924 13:42:17.000072 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "83406e20fcb14c6ba675854f1e61b401"
format_stamp: "Formatted at 2026-09-24 13:42:16 on dist-test-slave-tfm9"
server_key: "182e87736c12707632a384946731ef41"
server_key_iv: "7dbd5ed117d47bcf5f0db65e368d777e"
server_key_version: "encryptionkey@0"
I20260924 13:42:17.001355 19919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:17.002137 19934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:17.002338 19919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:42:17.002427 19919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "83406e20fcb14c6ba675854f1e61b401"
format_stamp: "Formatted at 2026-09-24 13:42:16 on dist-test-slave-tfm9"
server_key: "182e87736c12707632a384946731ef41"
server_key_iv: "7dbd5ed117d47bcf5f0db65e368d777e"
server_key_version: "encryptionkey@0"
I20260924 13:42:17.002512 19919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:17.015674 19919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:17.017482 19919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:17.017645 19919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:17.017867 19919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:17.018239 19919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:17.018298 19919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:17.018332 19919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:17.018361 19919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:17.024073 19919 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:35319
I20260924 13:42:17.024127 20047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:35319 every 8 connection(s)
I20260924 13:42:17.024430 19919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:42:17 dist-test-slave-tfm9 krb5kdc[19678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257336, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:17.033214 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:17.025097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:57635 (local address 127.19.53.254:43229)
0924 13:42:17.025160 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:17.025162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:17.025358 (+ 196us) server_negotiation.cc:407] Connection header received
0924 13:42:17.025880 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:17.025882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:17.025894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:17.025982 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:17.026884 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:17.027400 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:17.028372 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:17.028580 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:17.028662 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:17.030674 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:17.030675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:17.030676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:17.030683 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:17.031887 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:17.032268 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:17.032269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:17.032269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:17.032333 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:17.032574 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:17.032575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:17.032576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:17.032690 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:17.032758 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:17.033020 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 13:42:17.033124 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:42:17.033344 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:17.025108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:43229 (local address 127.19.53.194:57635)
0924 13:42:17.025243 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:17.025274 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:17.025774 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:17.026056 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:17.026059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:17.026601 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:17.026779 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:17.026784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:17.027493 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:17.027494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:17.028278 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:17.028280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:17.028673 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:17.028674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:17.028792 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:17.029180 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:17.029188 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:17.030559 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:17.031976 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:17.031982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:17.031984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:17.032191 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:17.032397 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:17.032398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:17.032398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:17.032499 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:17.032760 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:17.032767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:17.032892 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 13:42:17.032989 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 13:42:17.033135 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260924 13:42:17.033753 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 19919
I20260924 13:42:17.033842 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:42:17.033861 20048 heartbeater.cc:347] Connected to a master server at 127.19.53.254:43229
I20260924 13:42:17.033967 20048 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:17.034055 19671 external_mini_cluster.cc:1428] Setting key 3204ad5946385a5c1889aebe4d1bc56b
I20260924 13:42:17.034150 20048 heartbeater.cc:511] Master 127.19.53.254:43229 requested a full tablet report, sending...
I20260924 13:42:17.034575 19722 ts_manager.cc:194] Registered new tserver with Master: 83406e20fcb14c6ba675854f1e61b401 (127.19.53.194:35319)
I20260924 13:42:17.035077 19722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:57635
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:42:17.056740 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:43229
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:17.135762 20053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:17.135988 20053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:17.136029 20053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:17.137194 20053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:17.137259 20053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:17.137305 20053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:17.138710 20053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:43229
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:17.138950 20053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:17.139168 20053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:17.144060 20058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:17.144083 20061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:17.144089 20059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:17.144277 20053 server_base.cc:1034] running on GCE node
I20260924 13:42:17.144474 20053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:17.144667 20053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:17.145860 20053 hybrid_clock.cc:648] HybridClock initialized: now 1790257337145851 us; error 38 us; skew 500 ppm
Sep 24 13:42:17 dist-test-slave-tfm9 krb5kdc[19678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257337, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:17.148589 20053 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:42:17.148989 20053 webserver.cc:460] Webserver started at http://127.19.53.195:41151/ using document root <none> and password file <none>
I20260924 13:42:17.149215 20053 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:17.149283 20053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:17.149437 20053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:17.150521 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "4791ee4e86ef4f8887cbace62ef8ed68"
format_stamp: "Formatted at 2026-09-24 13:42:17 on dist-test-slave-tfm9"
server_key: "44434544b6924b405909bdfe1fb7e683"
server_key_iv: "000a1c6ef412a690c927253b94f366f2"
server_key_version: "encryptionkey@0"
I20260924 13:42:17.150876 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "4791ee4e86ef4f8887cbace62ef8ed68"
format_stamp: "Formatted at 2026-09-24 13:42:17 on dist-test-slave-tfm9"
server_key: "44434544b6924b405909bdfe1fb7e683"
server_key_iv: "000a1c6ef412a690c927253b94f366f2"
server_key_version: "encryptionkey@0"
I20260924 13:42:17.152166 20053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:17.152906 20068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:17.153096 20053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:17.153201 20053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "4791ee4e86ef4f8887cbace62ef8ed68"
format_stamp: "Formatted at 2026-09-24 13:42:17 on dist-test-slave-tfm9"
server_key: "44434544b6924b405909bdfe1fb7e683"
server_key_iv: "000a1c6ef412a690c927253b94f366f2"
server_key_version: "encryptionkey@0"
I20260924 13:42:17.153272 20053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:17.162384 20053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:17.164120 20053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:17.164276 20053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:17.164496 20053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:17.164783 20053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:17.164835 20053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:17.164873 20053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:17.164901 20053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:17.170601 20053 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:40745
I20260924 13:42:17.170673 20181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:40745 every 8 connection(s)
I20260924 13:42:17.170972 20053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:42:17 dist-test-slave-tfm9 krb5kdc[19678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257337, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:17.179383 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:17.171692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:54719 (local address 127.19.53.254:43229)
0924 13:42:17.171734 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:17.171737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:17.171922 (+ 185us) server_negotiation.cc:407] Connection header received
0924 13:42:17.172345 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:17.172346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:17.172356 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:17.172410 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:17.173269 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:17.173755 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:17.174793 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:17.174969 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:17.175022 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:17.177133 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:17.177134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:17.177134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:17.177141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:17.178345 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:17.178678 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:17.178679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:17.178679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:17.178727 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:17.178915 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:17.178915 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:17.178915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:17.179005 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:17.179053 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:17.179268 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 13:42:17.179324 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:17.179515 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:17.171637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:43229 (local address 127.19.53.195:54719)
0924 13:42:17.171792 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:17.171831 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:17.172267 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:17.172467 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:17.172471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:17.172996 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:17.173188 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:17.173192 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:17.173840 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:17.173841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:17.174721 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:17.174723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:17.175065 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:17.175066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:17.175159 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:17.175650 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:17.175659 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:17.176993 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:17.178411 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:17.178416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:17.178418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:17.178616 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:17.178779 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:17.178779 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:17.178779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:17.178865 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:17.179058 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:17.179062 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:17.179157 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 13:42:17.179239 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 13:42:17.179340 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260924 13:42:17.179999 20182 heartbeater.cc:347] Connected to a master server at 127.19.53.254:43229
I20260924 13:42:17.180124 20182 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:17.180286 20182 heartbeater.cc:511] Master 127.19.53.254:43229 requested a full tablet report, sending...
I20260924 13:42:17.180683 19722 ts_manager.cc:194] Registered new tserver with Master: 4791ee4e86ef4f8887cbace62ef8ed68 (127.19.53.195:40745)
I20260924 13:42:17.180998 19722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:54719
I20260924 13:42:17.181006 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 20053
I20260924 13:42:17.181095 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:42:17.181288 19671 external_mini_cluster.cc:1428] Setting key 6e696f6e9cb8616a732397d4359dcca9
I20260924 13:42:17.182354 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 13:42:17 dist-test-slave-tfm9 krb5kdc[19678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 13:42:17 dist-test-slave-tfm9 krb5kdc[19678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.53.193@KRBTEST.COM
I20260924 13:42:17.204253 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:17.196726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.19.53.193:35571)
0924 13:42:17.196907 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:17.196911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:17.196921 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:42:17.196947 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:17.196948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:17.196962 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:17.197025 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:17.197543 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:17.198327 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:17.199010 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:17.199305 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:17.199362 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:17.201458 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:17.201463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:17.201463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:17.201468 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:17.203087 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:17.203482 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:17.203485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:17.203486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:17.203564 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:17.203762 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:17.203763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:17.203763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:17.203883 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:17.203982 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:17.204053 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 13:42:17.204123 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":107,"threads_started":1}
W20260924 13:42:17.204617 19828 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54810
Sep 24 13:42:17 dist-test-slave-tfm9 krb5kdc[19678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 13:42:17 dist-test-slave-tfm9 krb5kdc[19678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.193@KRBTEST.COM
I20260924 13:42:17.227741 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:17.220062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.19.53.193:35571)
0924 13:42:17.220125 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:17.220128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:17.221113 (+ 985us) server_negotiation.cc:407] Connection header received
0924 13:42:17.221135 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:17.221136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:17.221144 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:17.221230 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:17.221722 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:17.222267 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:17.222953 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:17.223157 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:17.223263 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:17.225388 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:17.225389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:17.225390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:17.225398 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:17.226631 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:17.226962 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:17.226963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:17.226964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:17.227016 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:17.227227 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:17.227228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:17.227228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:17.227323 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:17.227448 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:17.227535 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 13:42:17.227637 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:17.228217 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 19785
I20260924 13:42:17.233258 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 19919
I20260924 13:42:17.237449 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 20053
I20260924 13:42:17.241354 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 19691
2026-09-24T13:42:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1829 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:17 dist-test-slave-tfm9 krb5kdc[20196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:17 dist-test-slave-tfm9 krb5kdc[20196](info): set up 2 sockets
Sep 24 13:42:17 dist-test-slave-tfm9 krb5kdc[20196](info): commencing operation
krb5kdc: starting...
W20260924 13:42:18.266983 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:18 dist-test-slave-tfm9 krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:42:18.382807 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:39005
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:39005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:18.463510 20209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:18.463711 20209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:18.463732 20209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:18.464829 20209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:18.464862 20209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:18.464876 20209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:18.464888 20209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:18.464900 20209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:18.466284 20209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:39005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:39005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:18.466461 20209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:18.466632 20209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:18.471457 20214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:18.471483 20209 server_base.cc:1034] running on GCE node
W20260924 13:42:18.471442 20217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:18.471536 20215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:18.472028 20209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:18.472365 20209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:18.473533 20209 hybrid_clock.cc:648] HybridClock initialized: now 1790257338473538 us; error 65 us; skew 500 ppm
Sep 24 13:42:18 dist-test-slave-tfm9 krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257338, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:18.476123 20209 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:42:18.476480 20209 webserver.cc:460] Webserver started at http://127.19.53.254:42599/ using document root <none> and password file <none>
I20260924 13:42:18.476727 20209 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:18.476771 20209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:18.476905 20209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:18.477901 20209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "516846a2534b49f6ac110042185317df"
format_stamp: "Formatted at 2026-09-24 13:42:18 on dist-test-slave-tfm9"
server_key: "9703f583a3858534788956b08f601b6b"
server_key_iv: "79ae96c243f7aab45b4d421a18681407"
server_key_version: "encryptionkey@0"
I20260924 13:42:18.478330 20209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "516846a2534b49f6ac110042185317df"
format_stamp: "Formatted at 2026-09-24 13:42:18 on dist-test-slave-tfm9"
server_key: "9703f583a3858534788956b08f601b6b"
server_key_iv: "79ae96c243f7aab45b4d421a18681407"
server_key_version: "encryptionkey@0"
I20260924 13:42:18.479655 20209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:18.480409 20224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:18.480625 20209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:18.480727 20209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "516846a2534b49f6ac110042185317df"
format_stamp: "Formatted at 2026-09-24 13:42:18 on dist-test-slave-tfm9"
server_key: "9703f583a3858534788956b08f601b6b"
server_key_iv: "79ae96c243f7aab45b4d421a18681407"
server_key_version: "encryptionkey@0"
I20260924 13:42:18.480804 20209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:18.508862 20209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:18.510968 20209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:18.511145 20209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:18.514956 20209 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:39005
I20260924 13:42:18.514992 20276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:39005 every 8 connection(s)
I20260924 13:42:18.515377 20209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:18.516222 20277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:18.517498 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 20209
I20260924 13:42:18.517598 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:18.517769 19671 external_mini_cluster.cc:1428] Setting key bd29dfa989afaf1e52a37c9aa54a3141
I20260924 13:42:18.519009 20277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df: Bootstrap starting.
I20260924 13:42:18.519671 20277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:18.519987 20277 log.cc:824] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:18.520740 20277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df: No bootstrap required, opened a new log
I20260924 13:42:18.522912 20277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516846a2534b49f6ac110042185317df" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39005 } }
I20260924 13:42:18.523109 20277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:18.523147 20277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516846a2534b49f6ac110042185317df, State: Initialized, Role: FOLLOWER
I20260924 13:42:18.523283 20277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516846a2534b49f6ac110042185317df" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39005 } }
I20260924 13:42:18.523396 20277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:18.523442 20277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:18.523494 20277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:18.523990 20277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516846a2534b49f6ac110042185317df" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39005 } }
I20260924 13:42:18.524113 20277 leader_election.cc:302] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 516846a2534b49f6ac110042185317df; no voters:
I20260924 13:42:18.524289 20277 leader_election.cc:288] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:18.524314 20282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:18.524420 20282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [term 1 LEADER]: Becoming Leader. State: Replica: 516846a2534b49f6ac110042185317df, State: Running, Role: LEADER
I20260924 13:42:18.524498 20282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516846a2534b49f6ac110042185317df" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39005 } }
I20260924 13:42:18.524540 20277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 13:42:18 dist-test-slave-tfm9 krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:18.524835 20282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "516846a2534b49f6ac110042185317df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516846a2534b49f6ac110042185317df" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39005 } } }
I20260924 13:42:18.524911 20282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:18.525034 20282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 516846a2534b49f6ac110042185317df. Latest consensus state: current_term: 1 leader_uuid: "516846a2534b49f6ac110042185317df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516846a2534b49f6ac110042185317df" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39005 } } }
I20260924 13:42:18.525102 20282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:18.525961 20288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:18.526549 20288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:18.527984 20288 catalog_manager.cc:1347] Generated new cluster ID: 9e0408488b8c4f0093fd2b4a78e1702b
I20260924 13:42:18.528036 20288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:18.529539 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:18.518331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.19.53.254:39005)
0924 13:42:18.518528 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:18.518538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:18.518574 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:42:18.519111 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:18.519120 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:18.519178 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:18.519334 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:18.519916 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:18.521268 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:18.522040 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:18.522745 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:18.522858 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:18.525374 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:18.525378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:18.525378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:18.525390 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:18.527672 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:18.528093 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:18.528096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:18.528096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:18.528229 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:18.528428 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:18.528428 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:18.528429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:18.528702 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:18.528829 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:18.529220 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 13:42:18.529342 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260924 13:42:18.533486 20288 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:18.534108 20288 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:18.542115 20288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 516846a2534b49f6ac110042185317df: Generated new TSK 0
I20260924 13:42:18.542295 20288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:42:18.572525 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:39005
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:18.651862 20303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:18.652043 20303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:18.652073 20303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:18.653182 20303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:18.653221 20303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:18.653275 20303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:18.654774 20303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:39005
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:18.655004 20303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:18.655246 20303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:18.660259 20311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:18.660251 20308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:18.660302 20309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:18.660607 20303 server_base.cc:1034] running on GCE node
I20260924 13:42:18.660816 20303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:18.661001 20303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:18.662141 20303 hybrid_clock.cc:648] HybridClock initialized: now 1790257338662118 us; error 43 us; skew 500 ppm
Sep 24 13:42:18 dist-test-slave-tfm9 krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257338, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:18.664505 20303 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:42:18.664909 20303 webserver.cc:460] Webserver started at http://127.19.53.193:41311/ using document root <none> and password file <none>
I20260924 13:42:18.665158 20303 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:18.665244 20303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:18.665388 20303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:18.666457 20303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "d09cf9615b9e4b2296695cf7417d14d8"
format_stamp: "Formatted at 2026-09-24 13:42:18 on dist-test-slave-tfm9"
server_key: "f56217a2edb2557ddee4d8b1c6246027"
server_key_iv: "c2098c945f5219a120fad334a74f4d6c"
server_key_version: "encryptionkey@0"
I20260924 13:42:18.666805 20303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "d09cf9615b9e4b2296695cf7417d14d8"
format_stamp: "Formatted at 2026-09-24 13:42:18 on dist-test-slave-tfm9"
server_key: "f56217a2edb2557ddee4d8b1c6246027"
server_key_iv: "c2098c945f5219a120fad334a74f4d6c"
server_key_version: "encryptionkey@0"
I20260924 13:42:18.668088 20303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:18.668812 20318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:18.668969 20303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:18.669067 20303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "d09cf9615b9e4b2296695cf7417d14d8"
format_stamp: "Formatted at 2026-09-24 13:42:18 on dist-test-slave-tfm9"
server_key: "f56217a2edb2557ddee4d8b1c6246027"
server_key_iv: "c2098c945f5219a120fad334a74f4d6c"
server_key_version: "encryptionkey@0"
I20260924 13:42:18.669142 20303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:18.703665 20303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:18.705571 20303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:18.705729 20303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:18.705981 20303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:18.706295 20303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:18.706348 20303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:18.706389 20303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:18.706420 20303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:18.711890 20303 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:34223
I20260924 13:42:18.711969 20431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:34223 every 8 connection(s)
I20260924 13:42:18.712296 20303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:42:18.717176 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 20303
I20260924 13:42:18.717249 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:42:18.717424 19671 external_mini_cluster.cc:1428] Setting key df483d88c7987f57f4cef29bec0e4a0d
Sep 24 13:42:18 dist-test-slave-tfm9 krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257338, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:18.721777 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:18.713049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:41321 (local address 127.19.53.254:39005)
0924 13:42:18.713098 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:18.713100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:18.713334 (+ 234us) server_negotiation.cc:407] Connection header received
0924 13:42:18.714245 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:18.714247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:18.714257 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:18.714308 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:18.714871 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:18.715364 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:18.716318 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:18.716510 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:18.716559 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:18.718752 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:18.718753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:18.718754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:18.718761 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:18.720435 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:18.720840 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:18.720841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:18.720842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:18.720895 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:18.721145 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:18.721145 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:18.721146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:18.721255 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:18.721327 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:18.721601 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 13:42:18.721686 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:42:18.721994 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:18.713026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:39005 (local address 127.19.53.193:41321)
0924 13:42:18.713214 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:18.713250 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:18.714155 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:18.714359 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:18.714362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:18.714607 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:18.714789 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:18.714793 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:18.715442 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:18.715443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:18.716245 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:18.716247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:18.716581 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:18.716581 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:18.716676 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:18.717130 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:18.717138 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:18.718656 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:18.720536 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:18.720542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:18.720544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:18.720754 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:18.720967 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:18.720968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:18.720968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:18.721073 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:18.721318 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:18.721324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:18.721472 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 13:42:18.721581 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 13:42:18.721739 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":110,"threads_started":1}
I20260924 13:42:18.722442 20432 heartbeater.cc:347] Connected to a master server at 127.19.53.254:39005
I20260924 13:42:18.722559 20432 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:18.722757 20432 heartbeater.cc:511] Master 127.19.53.254:39005 requested a full tablet report, sending...
I20260924 13:42:18.723310 20241 ts_manager.cc:194] Registered new tserver with Master: d09cf9615b9e4b2296695cf7417d14d8 (127.19.53.193:34223)
I20260924 13:42:18.724007 20241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:41321
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:42:18.742235 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:39005
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:18.824190 20437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:18.824371 20437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:18.824440 20437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:18.825701 20437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:18.825739 20437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:18.825766 20437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:18.827234 20437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:39005
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:18.827425 20437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:18.827592 20437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:18.832428 20443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:18.832479 20445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:18.832437 20442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:18.832768 20437 server_base.cc:1034] running on GCE node
I20260924 13:42:18.832979 20437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:18.833201 20437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:18.834371 20437 hybrid_clock.cc:648] HybridClock initialized: now 1790257338834356 us; error 27 us; skew 500 ppm
Sep 24 13:42:18 dist-test-slave-tfm9 krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257338, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:18.837074 20437 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:42:18.837538 20437 webserver.cc:460] Webserver started at http://127.19.53.194:38971/ using document root <none> and password file <none>
I20260924 13:42:18.837775 20437 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:18.837842 20437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:18.838004 20437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:18.839010 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "cd456542f5b64a67bd9b66d9da20729d"
format_stamp: "Formatted at 2026-09-24 13:42:18 on dist-test-slave-tfm9"
server_key: "c6afbe423cca78baa6f14a59b5cb3f47"
server_key_iv: "072f5785ca34e56d339d9d3ce62157a4"
server_key_version: "encryptionkey@0"
I20260924 13:42:18.839351 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "cd456542f5b64a67bd9b66d9da20729d"
format_stamp: "Formatted at 2026-09-24 13:42:18 on dist-test-slave-tfm9"
server_key: "c6afbe423cca78baa6f14a59b5cb3f47"
server_key_iv: "072f5785ca34e56d339d9d3ce62157a4"
server_key_version: "encryptionkey@0"
I20260924 13:42:18.840719 20437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:18.841463 20452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:18.841660 20437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:18.841756 20437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "cd456542f5b64a67bd9b66d9da20729d"
format_stamp: "Formatted at 2026-09-24 13:42:18 on dist-test-slave-tfm9"
server_key: "c6afbe423cca78baa6f14a59b5cb3f47"
server_key_iv: "072f5785ca34e56d339d9d3ce62157a4"
server_key_version: "encryptionkey@0"
I20260924 13:42:18.841842 20437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:18.855151 20437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:18.857353 20437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:18.857455 20437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:18.857613 20437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:18.857860 20437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:18.857888 20437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:18.857999 20437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:18.858026 20437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:18.863768 20437 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:44233
I20260924 13:42:18.863857 20565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:44233 every 8 connection(s)
I20260924 13:42:18.864138 20437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:42:18.866314 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 20437
I20260924 13:42:18.866387 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:42:18.866552 19671 external_mini_cluster.cc:1428] Setting key ec85946816e052908cdb60739fe1156d
Sep 24 13:42:18 dist-test-slave-tfm9 krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257338, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:18.874583 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:18.864874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:53849 (local address 127.19.53.254:39005)
0924 13:42:18.864935 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:18.864937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:18.865106 (+ 169us) server_negotiation.cc:407] Connection header received
0924 13:42:18.866061 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:18.866061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:18.866071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:18.866137 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:18.866846 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:18.867418 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:18.868490 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:18.868708 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:18.868766 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:18.871564 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:18.871565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:18.871566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:18.871571 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:18.873093 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:18.873582 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:18.873582 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:18.873583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:18.873642 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:18.873954 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:18.873955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:18.873955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:18.874066 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:18.874118 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:18.874457 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 13:42:18.874519 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 13:42:18.874846 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:18.864841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:39005 (local address 127.19.53.194:53849)
0924 13:42:18.864988 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:18.865028 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:18.865961 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:18.866184 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:18.866187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:18.866488 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:18.866769 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:18.866776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:18.867515 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:18.867516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:18.868390 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:18.868393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:18.868790 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:18.868791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:18.868944 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:18.869452 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:18.869462 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:18.871476 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:18.873177 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:18.873183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:18.873186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:18.873499 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:18.873706 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:18.873707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:18.873707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:18.873849 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:18.874138 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:18.874145 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:18.874312 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 13:42:18.874438 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 13:42:18.874615 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
I20260924 13:42:18.875339 20566 heartbeater.cc:347] Connected to a master server at 127.19.53.254:39005
I20260924 13:42:18.875429 20566 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:18.875614 20566 heartbeater.cc:511] Master 127.19.53.254:39005 requested a full tablet report, sending...
I20260924 13:42:18.875975 20241 ts_manager.cc:194] Registered new tserver with Master: cd456542f5b64a67bd9b66d9da20729d (127.19.53.194:44233)
I20260924 13:42:18.876375 20241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:53849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:42:18.892705 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:39005
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:18.976166 20571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:18.976406 20571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:18.976461 20571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:18.977628 20571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:18.977697 20571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:18.977746 20571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:18.979282 20571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:39005
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:18.979561 20571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:18.979782 20571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:18.984789 20577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:18.984953 20571 server_base.cc:1034] running on GCE node
W20260924 13:42:18.984783 20576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:18.984797 20579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:18.985288 20571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:18.985532 20571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:18.986682 20571 hybrid_clock.cc:648] HybridClock initialized: now 1790257338986671 us; error 26 us; skew 500 ppm
Sep 24 13:42:18 dist-test-slave-tfm9 krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257338, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:18.989163 20571 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:42:18.989648 20571 webserver.cc:460] Webserver started at http://127.19.53.195:35273/ using document root <none> and password file <none>
I20260924 13:42:18.989868 20571 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:18.989959 20571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:18.990091 20571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:18.991083 20571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "7c8559ea0b604568b5174666d6a6e446"
format_stamp: "Formatted at 2026-09-24 13:42:18 on dist-test-slave-tfm9"
server_key: "a51dd4ece04da86e74887df4d24a82f0"
server_key_iv: "0db9f2372ff0742ed3c5b2a2ab8d1703"
server_key_version: "encryptionkey@0"
I20260924 13:42:18.991425 20571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "7c8559ea0b604568b5174666d6a6e446"
format_stamp: "Formatted at 2026-09-24 13:42:18 on dist-test-slave-tfm9"
server_key: "a51dd4ece04da86e74887df4d24a82f0"
server_key_iv: "0db9f2372ff0742ed3c5b2a2ab8d1703"
server_key_version: "encryptionkey@0"
I20260924 13:42:18.992689 20571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:18.993342 20586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:18.993526 20571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:18.993618 20571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "7c8559ea0b604568b5174666d6a6e446"
format_stamp: "Formatted at 2026-09-24 13:42:18 on dist-test-slave-tfm9"
server_key: "a51dd4ece04da86e74887df4d24a82f0"
server_key_iv: "0db9f2372ff0742ed3c5b2a2ab8d1703"
server_key_version: "encryptionkey@0"
I20260924 13:42:18.993697 20571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:19.012918 20571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:19.014811 20571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:19.014966 20571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:19.015206 20571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:19.015514 20571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:19.015564 20571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:19.015597 20571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:19.015625 20571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:19.021397 20571 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:43059
I20260924 13:42:19.021442 20699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:43059 every 8 connection(s)
I20260924 13:42:19.021781 20571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:42:19.027419 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 20571
I20260924 13:42:19.027568 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:42:19.027750 19671 external_mini_cluster.cc:1428] Setting key 8f37fec6ca6782445ea257def860a8da
Sep 24 13:42:19 dist-test-slave-tfm9 krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257338, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:19.031448 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.022487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:46227 (local address 127.19.53.254:39005)
0924 13:42:19.022558 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:19.022560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:19.022728 (+ 168us) server_negotiation.cc:407] Connection header received
0924 13:42:19.023628 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:19.023629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:19.023637 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:19.023701 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:19.024261 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.024814 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.025956 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.026209 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.026262 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.028656 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:19.028657 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:19.028658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:19.028668 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:19.030129 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:19.030535 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:19.030537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:19.030538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:19.030593 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:19.030828 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:19.030829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:19.030829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:19.030942 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:19.031014 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:19.031274 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 13:42:19.031359 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:42:19.031667 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.022455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:39005 (local address 127.19.53.195:46227)
0924 13:42:19.022608 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:19.022647 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:19.023541 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:19.023755 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:19.023758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:19.024008 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:19.024194 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:19.024198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.024891 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.024892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:19.025854 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:19.025857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.026434 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.026435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:19.026540 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.026945 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:19.026953 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:19.028552 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:19.030231 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:19.030237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:19.030239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:19.030454 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:19.030655 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:19.030656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:19.030656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:19.030765 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:19.031004 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:19.031010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:19.031149 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 13:42:19.031253 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 13:42:19.031413 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260924 13:42:19.032065 20700 heartbeater.cc:347] Connected to a master server at 127.19.53.254:39005
I20260924 13:42:19.032155 20700 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:19.032313 20700 heartbeater.cc:511] Master 127.19.53.254:39005 requested a full tablet report, sending...
I20260924 13:42:19.032647 20241 ts_manager.cc:194] Registered new tserver with Master: 7c8559ea0b604568b5174666d6a6e446 (127.19.53.195:43059)
I20260924 13:42:19.032959 20241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:46227
I20260924 13:42:19.039381 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:42:19.046418 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.040885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.19.53.254:39005)
0924 13:42:19.040934 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:19.040936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:19.041037 (+ 101us) server_negotiation.cc:407] Connection header received
0924 13:42:19.041074 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:19.041075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:19.041083 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:19.041126 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:19.041676 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.042241 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.042947 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.043116 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.043181 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.044161 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:19.044162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:19.044162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:19.044168 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:19.045457 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:19.045777 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:19.045778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:19.045778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:19.045823 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:19.046012 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:19.046013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:19.046013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:19.046138 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:19.046212 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:19.046294 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 13:42:19.046345 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 13:42:19.048331 20241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:19.049831 20241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:19.056031 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.051973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:34223 (local address 127.0.0.1:37400)
0924 13:42:19.052432 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:19.052443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:19.052508 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:19.052776 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:19.052777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:19.052779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:19.052828 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:19.052829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.054138 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.054139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:19.055019 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:19.055021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.055807 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.055808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:19.055867 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.055869 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:42:19.055921 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 13:42:19.055952 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":24,"thread_start_us":70,"threads_started":1}
I20260924 13:42:19.056082 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.052157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.19.53.193:34223)
0924 13:42:19.052348 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:19.052353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:19.052524 (+ 171us) server_negotiation.cc:407] Connection header received
0924 13:42:19.052615 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:19.052618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:19.052635 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:19.052905 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:19.052992 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.054037 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.055096 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.055726 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.055806 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.055860 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:19.055938 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 13:42:19.055979 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260924 13:42:19.056133 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.052174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.19.53.194:44233)
0924 13:42:19.052354 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:19.052357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:19.052368 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:42:19.052452 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:19.052455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:19.052473 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:19.052551 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:19.052793 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.053537 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.054941 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.055589 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.055666 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.055726 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:19.055992 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 13:42:19.056039 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":73,"threads_started":1}
I20260924 13:42:19.056447 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.052180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.19.53.195:43059)
0924 13:42:19.052347 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:19.052352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:19.052593 (+ 241us) server_negotiation.cc:407] Connection header received
0924 13:42:19.052632 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:19.052634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:19.052647 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:19.052723 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:19.052906 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.054006 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.055339 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.056029 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.056094 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.056147 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:19.056312 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 13:42:19.056351 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260924 13:42:19.056363 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.051990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:43059 (local address 127.0.0.1:54364)
0924 13:42:19.052468 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:19.052475 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:19.052536 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:19.052729 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:19.052730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:19.052732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:19.052767 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:19.052769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.054113 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.054114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:19.055263 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:19.055265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.056184 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.056185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:19.056244 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.056248 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:19.056292 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 13:42:19.056319 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":27,"thread_start_us":85,"threads_started":1}
I20260924 13:42:19.056669 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.051973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:44233 (local address 127.0.0.1:43014)
0924 13:42:19.052233 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:19.052273 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:19.052340 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:19.052633 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:19.052636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:19.052638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:19.052710 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:19.052711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.053635 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.053636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:19.054849 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:19.054851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.055676 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.055677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:19.055754 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.055767 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 13:42:19.055833 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 13:42:19.055899 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":59,"threads_started":1}
I20260924 13:42:19.057193 20634 tablet_service.cc:1467] Processing CreateTablet for tablet 01548ccdcc67440d8c96d6c8b2b3c83c (DEFAULT_TABLE table=test-table [id=e3f38efbb7394ebcafc488fc1facecfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:19.057188 20366 tablet_service.cc:1467] Processing CreateTablet for tablet 01548ccdcc67440d8c96d6c8b2b3c83c (DEFAULT_TABLE table=test-table [id=e3f38efbb7394ebcafc488fc1facecfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:19.057264 20500 tablet_service.cc:1467] Processing CreateTablet for tablet 01548ccdcc67440d8c96d6c8b2b3c83c (DEFAULT_TABLE table=test-table [id=e3f38efbb7394ebcafc488fc1facecfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:19.057454 20634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01548ccdcc67440d8c96d6c8b2b3c83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:19.057508 20366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01548ccdcc67440d8c96d6c8b2b3c83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:19.057512 20500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01548ccdcc67440d8c96d6c8b2b3c83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:19.060135 20717 tablet_bootstrap.cc:492] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446: Bootstrap starting.
I20260924 13:42:19.060686 20718 tablet_bootstrap.cc:492] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8: Bootstrap starting.
I20260924 13:42:19.060940 20719 tablet_bootstrap.cc:492] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d: Bootstrap starting.
I20260924 13:42:19.061203 20717 tablet_bootstrap.cc:654] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:19.061321 20718 tablet_bootstrap.cc:654] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:19.061560 20719 tablet_bootstrap.cc:654] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:19.061842 20717 log.cc:824] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:19.061944 20718 log.cc:824] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:19.062111 20719 log.cc:824] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:19.062634 20718 tablet_bootstrap.cc:492] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8: No bootstrap required, opened a new log
I20260924 13:42:19.062665 20717 tablet_bootstrap.cc:492] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446: No bootstrap required, opened a new log
I20260924 13:42:19.062712 20718 ts_tablet_manager.cc:1397] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:42:19.062724 20717 ts_tablet_manager.cc:1397] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 13:42:19.062834 20719 tablet_bootstrap.cc:492] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d: No bootstrap required, opened a new log
I20260924 13:42:19.062892 20719 ts_tablet_manager.cc:1397] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 13:42:19.064085 20718 raft_consensus.cc:348] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd456542f5b64a67bd9b66d9da20729d" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44233 } } peers { permanent_uuid: "d09cf9615b9e4b2296695cf7417d14d8" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34223 } } peers { permanent_uuid: "7c8559ea0b604568b5174666d6a6e446" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43059 } }
I20260924 13:42:19.064348 20718 raft_consensus.cc:374] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:19.064481 20718 raft_consensus.cc:729] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09cf9615b9e4b2296695cf7417d14d8, State: Initialized, Role: FOLLOWER
I20260924 13:42:19.064641 20718 consensus_queue.cc:260] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd456542f5b64a67bd9b66d9da20729d" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44233 } } peers { permanent_uuid: "d09cf9615b9e4b2296695cf7417d14d8" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34223 } } peers { permanent_uuid: "7c8559ea0b604568b5174666d6a6e446" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43059 } }
I20260924 13:42:19.064708 20717 raft_consensus.cc:348] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd456542f5b64a67bd9b66d9da20729d" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44233 } } peers { permanent_uuid: "d09cf9615b9e4b2296695cf7417d14d8" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34223 } } peers { permanent_uuid: "7c8559ea0b604568b5174666d6a6e446" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43059 } }
I20260924 13:42:19.064883 20717 raft_consensus.cc:374] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:19.064924 20717 raft_consensus.cc:729] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8559ea0b604568b5174666d6a6e446, State: Initialized, Role: FOLLOWER
I20260924 13:42:19.064942 20718 ts_tablet_manager.cc:1428] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:42:19.064845 20719 raft_consensus.cc:348] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd456542f5b64a67bd9b66d9da20729d" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44233 } } peers { permanent_uuid: "d09cf9615b9e4b2296695cf7417d14d8" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34223 } } peers { permanent_uuid: "7c8559ea0b604568b5174666d6a6e446" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43059 } }
I20260924 13:42:19.064990 20719 raft_consensus.cc:374] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:19.065047 20719 raft_consensus.cc:729] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd456542f5b64a67bd9b66d9da20729d, State: Initialized, Role: FOLLOWER
I20260924 13:42:19.065066 20432 heartbeater.cc:503] Master 127.19.53.254:39005 was elected leader, sending a full tablet report...
I20260924 13:42:19.065037 20717 consensus_queue.cc:260] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd456542f5b64a67bd9b66d9da20729d" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44233 } } peers { permanent_uuid: "d09cf9615b9e4b2296695cf7417d14d8" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34223 } } peers { permanent_uuid: "7c8559ea0b604568b5174666d6a6e446" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43059 } }
I20260924 13:42:19.065174 20719 consensus_queue.cc:260] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd456542f5b64a67bd9b66d9da20729d" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44233 } } peers { permanent_uuid: "d09cf9615b9e4b2296695cf7417d14d8" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34223 } } peers { permanent_uuid: "7c8559ea0b604568b5174666d6a6e446" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43059 } }
I20260924 13:42:19.065318 20717 ts_tablet_manager.cc:1428] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 13:42:19.065418 20719 ts_tablet_manager.cc:1428] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 13:42:19.065529 20566 heartbeater.cc:503] Master 127.19.53.254:39005 was elected leader, sending a full tablet report...
I20260924 13:42:19.065691 20700 heartbeater.cc:503] Master 127.19.53.254:39005 was elected leader, sending a full tablet report...
W20260924 13:42:19.114681 20567 tablet.cc:2369] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:42:19.213032 20433 tablet.cc:2369] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:42:19.257855 20724 raft_consensus.cc:482] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:19.258067 20724 raft_consensus.cc:504] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd456542f5b64a67bd9b66d9da20729d" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44233 } } peers { permanent_uuid: "d09cf9615b9e4b2296695cf7417d14d8" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34223 } } peers { permanent_uuid: "7c8559ea0b604568b5174666d6a6e446" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43059 } }
I20260924 13:42:19.258447 20724 leader_election.cc:288] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd456542f5b64a67bd9b66d9da20729d (127.19.53.194:44233), d09cf9615b9e4b2296695cf7417d14d8 (127.19.53.193:34223)
I20260924 13:42:19.261173 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.258672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:60679 (local address 127.19.53.194:44233)
0924 13:42:19.258714 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:19.258716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:19.258741 (+ 25us) server_negotiation.cc:407] Connection header received
0924 13:42:19.258812 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:19.258813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:19.258822 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:19.258879 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:19.259041 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.259542 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.260456 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.260829 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.260924 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.260933 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:19.261074 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 13:42:19.261113 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:19.261173 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.258629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:44233 (local address 127.19.53.195:60679)
0924 13:42:19.258687 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:19.258700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:19.258738 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:19.258883 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:19.258883 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:19.258885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:19.258961 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:19.258963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.259621 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.259623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:19.260368 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:19.260370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.260911 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.260912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:19.260995 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.260998 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:19.261066 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 13:42:19.261102 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 13:42:19.261462 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.258692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:34461 (local address 127.19.53.193:34223)
0924 13:42:19.258740 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:19.258743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:19.258877 (+ 134us) server_negotiation.cc:407] Connection header received
0924 13:42:19.258964 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:19.258965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:19.258974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:19.259051 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:19.259195 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.259676 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.260591 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.260979 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.261066 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.261073 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:19.261354 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 13:42:19.261397 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:42:19.261488 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.258639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:34223 (local address 127.19.53.195:34461)
0924 13:42:19.258810 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:19.258819 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:19.258892 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:19.259057 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:19.259058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:19.259059 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:19.259131 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:19.259133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.259757 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.259758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:19.260502 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:19.260504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.261067 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.261068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:19.261145 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.261149 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:19.261218 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 13:42:19.261417 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":115,"threads_started":1}
I20260924 13:42:19.261490 20520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01548ccdcc67440d8c96d6c8b2b3c83c" candidate_uuid: "7c8559ea0b604568b5174666d6a6e446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd456542f5b64a67bd9b66d9da20729d" is_pre_election: true
I20260924 13:42:19.261658 20520 raft_consensus.cc:2454] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8559ea0b604568b5174666d6a6e446 in term 0.
I20260924 13:42:19.261814 20386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01548ccdcc67440d8c96d6c8b2b3c83c" candidate_uuid: "7c8559ea0b604568b5174666d6a6e446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09cf9615b9e4b2296695cf7417d14d8" is_pre_election: true
I20260924 13:42:19.261896 20587 leader_election.cc:302] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8559ea0b604568b5174666d6a6e446, cd456542f5b64a67bd9b66d9da20729d; no voters:
I20260924 13:42:19.261996 20386 raft_consensus.cc:2454] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8559ea0b604568b5174666d6a6e446 in term 0.
I20260924 13:42:19.262032 20724 raft_consensus.cc:2781] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:19.262084 20724 raft_consensus.cc:482] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:19.262106 20724 raft_consensus.cc:3037] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:19.262847 20724 raft_consensus.cc:504] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd456542f5b64a67bd9b66d9da20729d" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44233 } } peers { permanent_uuid: "d09cf9615b9e4b2296695cf7417d14d8" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34223 } } peers { permanent_uuid: "7c8559ea0b604568b5174666d6a6e446" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43059 } }
I20260924 13:42:19.262980 20724 leader_election.cc:288] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [CANDIDATE]: Term 1 election: Requested vote from peers cd456542f5b64a67bd9b66d9da20729d (127.19.53.194:44233), d09cf9615b9e4b2296695cf7417d14d8 (127.19.53.193:34223)
I20260924 13:42:19.263142 20520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01548ccdcc67440d8c96d6c8b2b3c83c" candidate_uuid: "7c8559ea0b604568b5174666d6a6e446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd456542f5b64a67bd9b66d9da20729d"
I20260924 13:42:19.263182 20386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01548ccdcc67440d8c96d6c8b2b3c83c" candidate_uuid: "7c8559ea0b604568b5174666d6a6e446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09cf9615b9e4b2296695cf7417d14d8"
I20260924 13:42:19.263232 20520 raft_consensus.cc:3037] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:19.263257 20386 raft_consensus.cc:3037] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:19.263909 20520 raft_consensus.cc:2454] T 01548ccdcc67440d8c96d6c8b2b3c83c P cd456542f5b64a67bd9b66d9da20729d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8559ea0b604568b5174666d6a6e446 in term 1.
I20260924 13:42:19.263926 20386 raft_consensus.cc:2454] T 01548ccdcc67440d8c96d6c8b2b3c83c P d09cf9615b9e4b2296695cf7417d14d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8559ea0b604568b5174666d6a6e446 in term 1.
I20260924 13:42:19.264071 20587 leader_election.cc:302] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8559ea0b604568b5174666d6a6e446, cd456542f5b64a67bd9b66d9da20729d; no voters:
I20260924 13:42:19.264166 20724 raft_consensus.cc:2781] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:19.264333 20724 raft_consensus.cc:686] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [term 1 LEADER]: Becoming Leader. State: Replica: 7c8559ea0b604568b5174666d6a6e446, State: Running, Role: LEADER
I20260924 13:42:19.264424 20724 consensus_queue.cc:237] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd456542f5b64a67bd9b66d9da20729d" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44233 } } peers { permanent_uuid: "d09cf9615b9e4b2296695cf7417d14d8" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34223 } } peers { permanent_uuid: "7c8559ea0b604568b5174666d6a6e446" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43059 } }
I20260924 13:42:19.265236 20241 catalog_manager.cc:5495] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c8559ea0b604568b5174666d6a6e446 (127.19.53.195). New cstate: current_term: 1 leader_uuid: "7c8559ea0b604568b5174666d6a6e446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd456542f5b64a67bd9b66d9da20729d" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d09cf9615b9e4b2296695cf7417d14d8" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8559ea0b604568b5174666d6a6e446" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43059 } health_report { overall_health: HEALTHY } } }
W20260924 13:42:19.272305 20701 tablet.cc:2369] T 01548ccdcc67440d8c96d6c8b2b3c83c P 7c8559ea0b604568b5174666d6a6e446: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 13:42:19 dist-test-slave-tfm9 krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.193@KRBTEST.COM
I20260924 13:42:19.285741 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.278683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.19.53.193:34223)
0924 13:42:19.278749 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:19.278751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:19.278821 (+ 70us) server_negotiation.cc:407] Connection header received
0924 13:42:19.278864 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:19.278865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:19.278874 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:19.278924 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:19.279437 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.279874 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.280557 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.280734 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.280787 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.283080 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:19.283090 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:19.283090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:19.283098 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:19.284738 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:19.285057 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:19.285059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:19.285060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:19.285115 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:19.285315 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:19.285315 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:19.285316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:19.285450 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:19.285572 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:19.285626 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 13:42:19.285683 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 13:42:19 dist-test-slave-tfm9 krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 13:42:19 dist-test-slave-tfm9 krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.53.193@KRBTEST.COM
I20260924 13:42:19.307757 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.300701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.19.53.193:34223)
0924 13:42:19.300770 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:19.300772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:19.300848 (+ 76us) server_negotiation.cc:407] Connection header received
0924 13:42:19.300935 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:19.300936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:19.300953 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:19.301017 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:19.301542 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.302065 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.302791 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.303009 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.303081 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.305202 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:19.305204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:19.305204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:19.305212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:19.306549 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:19.306892 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:19.306893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:19.306894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:19.306950 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:19.307153 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:19.307154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:19.307154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:19.307248 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:19.307346 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:19.307563 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 13:42:19.307643 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 13:42:19 dist-test-slave-tfm9 krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 13:42:19 dist-test-slave-tfm9 krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.193@KRBTEST.COM
I20260924 13:42:19.329137 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:19.322307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.19.53.193:34223)
0924 13:42:19.322387 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:19.322388 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:19.322446 (+ 58us) server_negotiation.cc:407] Connection header received
0924 13:42:19.322545 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:19.322546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:19.322555 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:19.322604 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:19.323162 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.323628 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.324372 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:19.324605 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:19.324671 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:19.327069 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:19.327071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:19.327071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:19.327077 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:19.328259 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:19.328568 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:19.328568 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:19.328569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:19.328615 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:19.328770 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:19.328771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:19.328771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:19.328854 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:19.328924 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:19.329021 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 13:42:19.329074 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 13:42:19.329614 20346 tablet_service.cc:2908] Rejecting scan request for tablet 01548ccdcc67440d8c96d6c8b2b3c83c: Uninitialized: clean time has not yet been initialized
I20260924 13:42:19.330439 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 20303
I20260924 13:42:19.336272 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 20437
I20260924 13:42:19.341077 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 20571
I20260924 13:42:19.345484 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 20209
2026-09-24T13:42:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2104 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:19 dist-test-slave-tfm9 krb5kdc[20741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:19 dist-test-slave-tfm9 krb5kdc[20741](info): set up 2 sockets
Sep 24 13:42:19 dist-test-slave-tfm9 krb5kdc[20741](info): commencing operation
krb5kdc: starting...
W20260924 13:42:20.378974 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:20 dist-test-slave-tfm9 krb5kdc[20741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:42:20.488639 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:36033
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:36033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:20.567693 20754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:20.567899 20754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:20.567921 20754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:20.568997 20754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:20.569037 20754 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 13:42:20.569057 20754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:20.569070 20754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:20.569083 20754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:20.569095 20754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:20.570487 20754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:36033
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:36033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:20.570667 20754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:20.570824 20754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:20.575534 20760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:20.575526 20759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:20.575526 20762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:20.575609 20754 server_base.cc:1034] running on GCE node
I20260924 13:42:20.575960 20754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:20.576253 20754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:20.577436 20754 hybrid_clock.cc:648] HybridClock initialized: now 1790257340577415 us; error 37 us; skew 500 ppm
Sep 24 13:42:20 dist-test-slave-tfm9 krb5kdc[20741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257340, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:20.589771 20754 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:42:20.590323 20754 webserver.cc:460] Webserver started at http://127.19.53.254:34913/ using document root <none> and password file <none>
I20260924 13:42:20.590509 20754 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:20.590569 20754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:20.590668 20754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:20.591687 20754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "4ad4813495024e9b9c4233c652f1e5c1"
format_stamp: "Formatted at 2026-09-24 13:42:20 on dist-test-slave-tfm9"
server_key: "b2c0ec61684a84ed5428427da109f540"
server_key_iv: "65dd924e2363e8e26a5ed0d459ea3817"
server_key_version: "encryptionkey@0"
I20260924 13:42:20.592033 20754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "4ad4813495024e9b9c4233c652f1e5c1"
format_stamp: "Formatted at 2026-09-24 13:42:20 on dist-test-slave-tfm9"
server_key: "b2c0ec61684a84ed5428427da109f540"
server_key_iv: "65dd924e2363e8e26a5ed0d459ea3817"
server_key_version: "encryptionkey@0"
I20260924 13:42:20.593385 20754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:20.594113 20769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:20.594290 20754 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:20.594370 20754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "4ad4813495024e9b9c4233c652f1e5c1"
format_stamp: "Formatted at 2026-09-24 13:42:20 on dist-test-slave-tfm9"
server_key: "b2c0ec61684a84ed5428427da109f540"
server_key_iv: "65dd924e2363e8e26a5ed0d459ea3817"
server_key_version: "encryptionkey@0"
I20260924 13:42:20.594422 20754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:20.616698 20754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:20.618494 20754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:20.618607 20754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:20.622759 20754 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:36033
I20260924 13:42:20.622864 20831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:36033 every 8 connection(s)
I20260924 13:42:20.623162 20754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:20.623308 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 20754
I20260924 13:42:20.623404 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:20.623584 19671 external_mini_cluster.cc:1428] Setting key 98eac64b4260aec77e0268578b23df6a
I20260924 13:42:20.623927 20832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:20.627168 20832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1: Bootstrap starting.
I20260924 13:42:20.627826 20832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:20.628134 20832 log.cc:824] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:20.628952 20832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1: No bootstrap required, opened a new log
Sep 24 13:42:20 dist-test-slave-tfm9 krb5kdc[20741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:20.631296 20832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ad4813495024e9b9c4233c652f1e5c1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 36033 } }
I20260924 13:42:20.631457 20832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:20.631494 20832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad4813495024e9b9c4233c652f1e5c1, State: Initialized, Role: FOLLOWER
I20260924 13:42:20.631641 20832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ad4813495024e9b9c4233c652f1e5c1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 36033 } }
I20260924 13:42:20.631707 20832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:20.631728 20832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:20.631814 20832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:20.632328 20832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ad4813495024e9b9c4233c652f1e5c1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 36033 } }
I20260924 13:42:20.632434 20832 leader_election.cc:302] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ad4813495024e9b9c4233c652f1e5c1; no voters:
I20260924 13:42:20.632615 20832 leader_election.cc:288] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:20.632668 20837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:20.632822 20837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 4ad4813495024e9b9c4233c652f1e5c1, State: Running, Role: LEADER
I20260924 13:42:20.632890 20832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:20.632947 20837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ad4813495024e9b9c4233c652f1e5c1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 36033 } }
I20260924 13:42:20.633313 20838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ad4813495024e9b9c4233c652f1e5c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ad4813495024e9b9c4233c652f1e5c1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 36033 } } }
I20260924 13:42:20.633301 20837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ad4813495024e9b9c4233c652f1e5c1. Latest consensus state: current_term: 1 leader_uuid: "4ad4813495024e9b9c4233c652f1e5c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ad4813495024e9b9c4233c652f1e5c1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 36033 } } }
I20260924 13:42:20.633394 20838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:20.633397 20837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:20.633680 20845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:20.634227 20845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:20.634627 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:20.624041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.19.53.254:36033)
0924 13:42:20.624214 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:20.624224 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:20.624255 (+ 31us) server_negotiation.cc:407] Connection header received
0924 13:42:20.624672 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:20.624678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:20.624719 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:20.624826 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:20.625549 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.626480 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.627515 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.627960 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.628042 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:20.630734 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:20.630738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:20.630738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:20.630747 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:20.632359 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:20.632814 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:20.632817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:20.632817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:20.632946 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:20.633178 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:20.633179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:20.633179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:20.633499 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:20.633672 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:20.634250 (+ 578us) server_negotiation.cc:299] Negotiation successful
0924 13:42:20.634400 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260924 13:42:20.635699 20845 catalog_manager.cc:1347] Generated new cluster ID: bf2e50a4d3a6463fb9d965016aca3327
I20260924 13:42:20.635747 20845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:20.649791 20845 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:20.650364 20845 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:20.661402 20845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1: Generated new TSK 0
I20260924 13:42:20.661595 20845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:42:20.694872 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:36033
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:20.775444 20858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:20.775641 20858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:20.775665 20858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:20.776746 20858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:20.776779 20858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:20.776799 20858 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:42:20.776815 20858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:20.778164 20858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.53.254:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:20.778347 20858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:20.778507 20858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:20.783283 20863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:20.783260 20866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:20.783500 20858 server_base.cc:1034] running on GCE node
W20260924 13:42:20.783301 20864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:20.783774 20858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:20.784022 20858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:20.785190 20858 hybrid_clock.cc:648] HybridClock initialized: now 1790257340785189 us; error 37 us; skew 500 ppm
Sep 24 13:42:20 dist-test-slave-tfm9 krb5kdc[20741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257340, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:20.787638 20858 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:42:20.788064 20858 webserver.cc:460] Webserver started at http://127.19.53.193:40705/ using document root <none> and password file <none>
I20260924 13:42:20.788281 20858 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:20.788343 20858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:20.788487 20858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:20.789491 20858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "0459a14838fb4be99334b36d74622e7d"
format_stamp: "Formatted at 2026-09-24 13:42:20 on dist-test-slave-tfm9"
server_key: "1963f7fdea2e6ab75772f94ec615ebf8"
server_key_iv: "2d15a32966859a66dc6b2fc182bb03c1"
server_key_version: "encryptionkey@0"
I20260924 13:42:20.789839 20858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "0459a14838fb4be99334b36d74622e7d"
format_stamp: "Formatted at 2026-09-24 13:42:20 on dist-test-slave-tfm9"
server_key: "1963f7fdea2e6ab75772f94ec615ebf8"
server_key_iv: "2d15a32966859a66dc6b2fc182bb03c1"
server_key_version: "encryptionkey@0"
I20260924 13:42:20.791167 20858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:20.792098 20873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:20.792281 20858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:20.792362 20858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "0459a14838fb4be99334b36d74622e7d"
format_stamp: "Formatted at 2026-09-24 13:42:20 on dist-test-slave-tfm9"
server_key: "1963f7fdea2e6ab75772f94ec615ebf8"
server_key_iv: "2d15a32966859a66dc6b2fc182bb03c1"
server_key_version: "encryptionkey@0"
I20260924 13:42:20.792412 20858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:20.801833 20858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:20.803505 20858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:20.803606 20858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:20.804000 20858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:20.804060 20858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:20.804082 20858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:20.804095 20858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:20.810973 20858 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:37537
I20260924 13:42:20.811043 20988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:37537 every 8 connection(s)
Sep 24 13:42:20 dist-test-slave-tfm9 krb5kdc[20741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257340, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:20.811327 20858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:42:20.813984 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:20.804476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:41935 (local address 127.19.53.254:36033)
0924 13:42:20.804518 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:20.804521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:20.804799 (+ 278us) server_negotiation.cc:407] Connection header received
0924 13:42:20.805281 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:20.805282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:20.805295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:20.805354 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:20.806313 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.807103 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.808716 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.809046 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.809115 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:20.811638 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:20.811639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:20.811640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:20.811647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:20.812900 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:20.813242 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:20.813243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:20.813243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:20.813292 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:20.813481 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:20.813482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:20.813482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:20.813577 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:20.813624 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:20.813816 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 13:42:20.813870 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:42:20.814183 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:20.804530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:36033 (local address 127.19.53.193:41935)
0924 13:42:20.804692 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:20.804722 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:20.805173 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:20.805454 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:20.805459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:20.805862 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:20.806206 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:20.806213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.807193 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.807194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:20.808623 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:20.808626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.809131 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.809132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:20.809296 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:20.809778 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:20.809796 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:20.811518 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:20.812969 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:20.812974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:20.812976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:20.813181 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:20.813344 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:20.813345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:20.813345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:20.813433 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:20.813625 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:20.813629 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:20.813731 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 13:42:20.813807 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 13:42:20.813964 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260924 13:42:20.814659 20989 heartbeater.cc:347] Connected to a master server at 127.19.53.254:36033
I20260924 13:42:20.814730 20989 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:20.814908 20989 heartbeater.cc:511] Master 127.19.53.254:36033 requested a full tablet report, sending...
I20260924 13:42:20.815399 20786 ts_manager.cc:194] Registered new tserver with Master: 0459a14838fb4be99334b36d74622e7d (127.19.53.193:37537)
I20260924 13:42:20.816306 20786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:41935
I20260924 13:42:20.819324 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 20858
I20260924 13:42:20.819417 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:42:20.819635 19671 external_mini_cluster.cc:1428] Setting key 3349ddd7c004409d7d58d364ec3fc1d2
I20260924 13:42:20.821200 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:20.815679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:34009 (local address 127.19.53.254:36033)
0924 13:42:20.815716 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:20.815718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:20.815827 (+ 109us) server_negotiation.cc:407] Connection header received
0924 13:42:20.815916 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:20.815917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:20.815924 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:20.815966 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:20.816289 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.816881 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.817576 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.817757 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.817825 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:20.818730 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:20.818732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:20.818732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:20.818738 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:20.820150 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:20.820561 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:20.820562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:20.820562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:20.820612 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:20.820784 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:20.820784 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:20.820784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:20.820874 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:20.820921 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:20.821074 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 13:42:20.821130 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:20.821406 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:20.815601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:36033 (local address 127.19.53.193:34009)
0924 13:42:20.815772 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:20.815783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:20.815838 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:20.815971 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:20.815972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:20.816167 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:20.816225 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:20.816227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.816973 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.816974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:20.817484 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:20.817487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.817852 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.817853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:20.817980 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:20.818373 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:20.818377 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:20.818656 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:20.820236 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:20.820238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:20.820238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:20.820492 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:20.820657 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:20.820658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:20.820658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:20.820726 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:20.820933 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:20.820934 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:20.820981 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 13:42:20.821051 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:42:20.821162 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:42:20.842607 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:36033
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:20.921787 20998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:20.922025 20998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:20.922052 20998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:20.923126 20998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:20.923159 20998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:20.923178 20998 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:42:20.923195 20998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:20.924520 20998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.53.254:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:20.924695 20998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:20.924852 20998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:20.929767 21004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:20.929741 21006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:20.929786 21003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:20.929894 20998 server_base.cc:1034] running on GCE node
I20260924 13:42:20.930117 20998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:20.930330 20998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:20.931468 20998 hybrid_clock.cc:648] HybridClock initialized: now 1790257340931448 us; error 33 us; skew 500 ppm
Sep 24 13:42:20 dist-test-slave-tfm9 krb5kdc[20741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257340, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:20.934119 20998 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:42:20.934496 20998 webserver.cc:460] Webserver started at http://127.19.53.194:35111/ using document root <none> and password file <none>
I20260924 13:42:20.934682 20998 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:20.934722 20998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:20.934808 20998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:20.935796 20998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "c7175f1bfbd34da8b00d9b640774dfe5"
format_stamp: "Formatted at 2026-09-24 13:42:20 on dist-test-slave-tfm9"
server_key: "b1f2af55ec0fe1c1807c9e570b57c537"
server_key_iv: "d0fba7b9785bca4c2fe39ba6fee56654"
server_key_version: "encryptionkey@0"
I20260924 13:42:20.936110 20998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "c7175f1bfbd34da8b00d9b640774dfe5"
format_stamp: "Formatted at 2026-09-24 13:42:20 on dist-test-slave-tfm9"
server_key: "b1f2af55ec0fe1c1807c9e570b57c537"
server_key_iv: "d0fba7b9785bca4c2fe39ba6fee56654"
server_key_version: "encryptionkey@0"
I20260924 13:42:20.937381 20998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:20.938068 21013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:20.938226 20998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:20.938320 20998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "c7175f1bfbd34da8b00d9b640774dfe5"
format_stamp: "Formatted at 2026-09-24 13:42:20 on dist-test-slave-tfm9"
server_key: "b1f2af55ec0fe1c1807c9e570b57c537"
server_key_iv: "d0fba7b9785bca4c2fe39ba6fee56654"
server_key_version: "encryptionkey@0"
I20260924 13:42:20.938396 20998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:20.946950 20998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:20.949184 20998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:20.949330 20998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:20.949806 20998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:20.949872 20998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:20.949961 20998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:20.950006 20998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:42:20 dist-test-slave-tfm9 krb5kdc[20741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257340, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:20.956833 20998 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:43981
I20260924 13:42:20.956908 21128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:43981 every 8 connection(s)
I20260924 13:42:20.957262 20998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:42:20.959323 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:20.950294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:45355 (local address 127.19.53.254:36033)
0924 13:42:20.950362 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:20.950365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:20.950569 (+ 204us) server_negotiation.cc:407] Connection header received
0924 13:42:20.951034 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:20.951036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:20.951049 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:20.951135 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:20.951979 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.952807 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.954266 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.954474 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.954555 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:20.956753 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:20.956754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:20.956754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:20.956762 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:20.958257 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:20.958624 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:20.958625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:20.958625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:20.958679 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:20.958870 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:20.958871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:20.958871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:20.958966 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:20.959013 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:20.959208 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 13:42:20.959262 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 13:42:20.959506 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:20.950288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:36033 (local address 127.19.53.194:45355)
0924 13:42:20.950437 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:20.950476 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:20.950947 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:20.951210 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:20.951229 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:20.951579 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:20.951877 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:20.951884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.952908 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.952909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:20.954180 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:20.954183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.954577 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.954578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:20.954696 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:20.955158 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:20.955166 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:20.956672 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:20.958323 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:20.958327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:20.958340 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:20.958561 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:20.958732 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:20.958733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:20.958734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:20.958822 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:20.959013 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:20.959016 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:20.959121 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 13:42:20.959196 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 13:42:20.959302 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260924 13:42:20.959928 21129 heartbeater.cc:347] Connected to a master server at 127.19.53.254:36033
I20260924 13:42:20.960007 21129 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:20.960182 21129 heartbeater.cc:511] Master 127.19.53.254:36033 requested a full tablet report, sending...
I20260924 13:42:20.960551 20785 ts_manager.cc:194] Registered new tserver with Master: c7175f1bfbd34da8b00d9b640774dfe5 (127.19.53.194:43981)
I20260924 13:42:20.960929 20785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:45355
I20260924 13:42:20.966363 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:20.961241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:40531 (local address 127.19.53.254:36033)
0924 13:42:20.961277 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:20.961279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:20.961763 (+ 484us) server_negotiation.cc:407] Connection header received
0924 13:42:20.961797 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:20.961798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:20.961806 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:20.961843 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:20.962170 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.962622 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.963261 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.963450 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.963510 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:20.964223 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:20.964224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:20.964224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:20.964228 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:20.965471 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:20.965731 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:20.965731 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:20.965732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:20.965778 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:20.965964 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:20.965964 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:20.965965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:20.966108 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:20.966157 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:20.966256 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 13:42:20.966307 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:42:20.966367 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:20.961167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:36033 (local address 127.19.53.194:40531)
0924 13:42:20.961697 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:20.961706 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:20.961753 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:20.961846 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:20.961846 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:20.962057 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:20.962114 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:20.962116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.962699 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.962700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:20.963179 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:20.963180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:20.963514 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:20.963515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:20.963564 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:20.963915 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:20.963919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:20.964161 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:20.965531 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:20.965531 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:20.965532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:20.965676 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:20.965838 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:20.965839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:20.965839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:20.965883 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:20.966156 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:20.966157 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:20.966191 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 13:42:20.966241 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:42:20.966308 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":480,"threads_started":1}
I20260924 13:42:20.966748 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 20998
I20260924 13:42:20.966843 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:42:20.967005 19671 external_mini_cluster.cc:1428] Setting key 9bd8857fc625cbebaa56b47d217def1d
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:42:20.988652 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:36033
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:21.068538 21138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:21.068745 21138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:21.068781 21138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:21.069886 21138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:21.069963 21138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:21.069996 21138 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:42:21.070013 21138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:21.071342 21138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.53.254:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:21.071511 21138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:21.071664 21138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:21.076375 21144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:21.076388 21143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:21.076375 21146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:21.076566 21138 server_base.cc:1034] running on GCE node
I20260924 13:42:21.076807 21138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:21.077024 21138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:21.078186 21138 hybrid_clock.cc:648] HybridClock initialized: now 1790257341078172 us; error 26 us; skew 500 ppm
Sep 24 13:42:21 dist-test-slave-tfm9 krb5kdc[20741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257341, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:21.080492 21138 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:42:21.080953 21138 webserver.cc:460] Webserver started at http://127.19.53.195:41871/ using document root <none> and password file <none>
I20260924 13:42:21.081180 21138 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:21.081243 21138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:21.081389 21138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:21.082412 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "575cd51400124f479060aa6907920c41"
format_stamp: "Formatted at 2026-09-24 13:42:21 on dist-test-slave-tfm9"
server_key: "da5806a5416a03e989dfe52b9a81171d"
server_key_iv: "76082fdbfcd587fc933ea1dc0805313c"
server_key_version: "encryptionkey@0"
I20260924 13:42:21.082768 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "575cd51400124f479060aa6907920c41"
format_stamp: "Formatted at 2026-09-24 13:42:21 on dist-test-slave-tfm9"
server_key: "da5806a5416a03e989dfe52b9a81171d"
server_key_iv: "76082fdbfcd587fc933ea1dc0805313c"
server_key_version: "encryptionkey@0"
I20260924 13:42:21.084065 21138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:21.084805 21153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:21.084978 21138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:21.085062 21138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "575cd51400124f479060aa6907920c41"
format_stamp: "Formatted at 2026-09-24 13:42:21 on dist-test-slave-tfm9"
server_key: "da5806a5416a03e989dfe52b9a81171d"
server_key_iv: "76082fdbfcd587fc933ea1dc0805313c"
server_key_version: "encryptionkey@0"
I20260924 13:42:21.085131 21138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:21.102807 21138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:21.104419 21138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:21.104569 21138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:21.105055 21138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:21.105106 21138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:21.105141 21138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:21.105165 21138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:42:21 dist-test-slave-tfm9 krb5kdc[20741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257341, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:21.112470 21138 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:39299
I20260924 13:42:21.112805 21138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:42:21.112802 21268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:39299 every 8 connection(s)
I20260924 13:42:21.114768 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.105420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:47817 (local address 127.19.53.254:36033)
0924 13:42:21.105482 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.105484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.105729 (+ 245us) server_negotiation.cc:407] Connection header received
0924 13:42:21.106327 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.106328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.106341 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.106412 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:21.106966 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.107548 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.108572 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.108765 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.108843 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.111795 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:21.111796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:21.111797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:21.111804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:21.113391 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:21.113790 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:21.113791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:21.113792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:21.113849 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:21.114122 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:21.114123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:21.114123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:21.114238 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:21.114309 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.114574 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.114678 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 13:42:21.114912 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.105410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:36033 (local address 127.19.53.195:47817)
0924 13:42:21.105607 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:21.105645 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:21.106216 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:21.106459 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:21.106462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:21.106704 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:21.106875 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.106879 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.107632 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.107633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.108453 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.108455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.108855 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.108855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.108999 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.109513 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:21.109524 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:21.111706 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:21.113482 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:21.113488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:21.113491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:21.113717 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:21.113932 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:21.113933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:21.113933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:21.114048 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:21.114300 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:21.114306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:21.114435 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 13:42:21.114531 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 13:42:21.114682 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":135,"threads_started":1}
I20260924 13:42:21.115345 21269 heartbeater.cc:347] Connected to a master server at 127.19.53.254:36033
I20260924 13:42:21.115435 21269 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:21.115600 21269 heartbeater.cc:511] Master 127.19.53.254:36033 requested a full tablet report, sending...
I20260924 13:42:21.115964 20785 ts_manager.cc:194] Registered new tserver with Master: 575cd51400124f479060aa6907920c41 (127.19.53.195:39299)
I20260924 13:42:21.116478 20785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:47817
I20260924 13:42:21.121692 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.116268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:36033 (local address 127.19.53.195:51007)
0924 13:42:21.116426 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:21.116437 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:21.116495 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:21.116644 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:21.116645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:21.116856 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:21.116908 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.116910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.117597 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.117598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.118215 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.118217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.118630 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.118630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.118682 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.119044 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:21.119048 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:21.119297 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:21.120836 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:21.120836 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:21.120837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:21.121007 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:21.121192 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:21.121193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:21.121193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:21.121251 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:21.121460 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:21.121460 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:21.121495 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:42:21.121554 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 13:42:21.121625 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
I20260924 13:42:21.121702 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.116351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:51007 (local address 127.19.53.254:36033)
0924 13:42:21.116389 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.116391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.116487 (+ 96us) server_negotiation.cc:407] Connection header received
0924 13:42:21.116564 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.116564 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.116575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.116626 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:21.116976 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.117517 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.118318 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.118559 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.118629 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.119422 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:21.119423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:21.119423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:21.119430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:21.120762 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:21.121070 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:21.121070 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:21.121071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:21.121120 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:21.121310 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:21.121311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:21.121311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:21.121406 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:21.121459 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.121584 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.121642 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:21.122866 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 21138
I20260924 13:42:21.122982 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:42:21.123201 19671 external_mini_cluster.cc:1428] Setting key f0722c8f6b4029c3a3f5cf01b0ab3d37
I20260924 13:42:21.123982 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 13:42:21 dist-test-slave-tfm9 krb5kdc[20741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257341, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 13:42:21 dist-test-slave-tfm9 krb5kdc[20741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257341, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:21.145454 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.138942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50214 (local address 127.19.53.254:36033)
0924 13:42:21.138980 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.138983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.138991 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:21.139066 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.139067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.139076 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.139132 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:21.139684 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.140305 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.141109 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.141307 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.141384 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.143519 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:21.143520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:21.143520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:21.143527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:21.144607 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:21.144890 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:21.144890 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:21.144891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:21.144936 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:21.145093 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:21.145094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:21.145094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:21.145181 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:21.145250 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.145325 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.145374 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:42:21.147082 20785 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:21.148730 20785 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:21.154856 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.150787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:37537 (local address 127.0.0.1:50286)
0924 13:42:21.151054 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:21.151065 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:21.151119 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:21.151625 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:21.151626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:21.151628 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:21.151678 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.151680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.152545 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.152546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.153873 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.153876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.154557 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.154559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.154638 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.154651 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 13:42:21.154716 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 13:42:21.154765 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":54,"thread_start_us":48,"threads_started":1}
I20260924 13:42:21.154991 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.151222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.19.53.194:43981)
0924 13:42:21.151494 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.151497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.151506 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:21.151539 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.151542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.151587 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.151654 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:21.151832 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.152809 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.153836 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.154647 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.154728 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.154777 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.154855 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.154894 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260924 13:42:21.154949 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.150785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:43981 (local address 127.0.0.1:57474)
0924 13:42:21.151211 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:21.151219 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:21.151254 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:21.151723 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:21.151724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:21.151726 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:21.151755 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.151756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.152885 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.152886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.153758 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.153760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.154732 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.154733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.154782 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.154785 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:21.154841 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 13:42:21.154876 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":3,"thread_start_us":50,"threads_started":1}
I20260924 13:42:21.155277 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.151016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.19.53.195:39299)
0924 13:42:21.151200 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.151205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.151214 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:21.151254 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.151257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.151272 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.151352 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:21.151545 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.152566 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.153988 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.154649 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.154725 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.154777 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.155120 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.155180 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260924 13:42:21.155469 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.150874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.19.53.193:37537)
0924 13:42:21.151446 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.151451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.151461 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:42:21.151502 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.151505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.151521 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.151755 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:21.151818 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.152461 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.154008 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.154472 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.154566 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.154628 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.155293 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.155380 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":98,"threads_started":1}
I20260924 13:42:21.155818 21063 tablet_service.cc:1467] Processing CreateTablet for tablet c77c03e85b7944cc86ba8cee4714198f (DEFAULT_TABLE table=test-table [id=272fc95d5248454dba2a531be0ed4a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:21.156205 21063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77c03e85b7944cc86ba8cee4714198f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:21.156134 20923 tablet_service.cc:1467] Processing CreateTablet for tablet c77c03e85b7944cc86ba8cee4714198f (DEFAULT_TABLE table=test-table [id=272fc95d5248454dba2a531be0ed4a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:21.156459 20923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77c03e85b7944cc86ba8cee4714198f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:21.155057 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.150785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:39299 (local address 127.0.0.1:50300)
0924 13:42:21.151022 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:21.151062 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:21.151111 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:21.151355 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:21.151358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:21.151359 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:21.151415 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.151417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.152639 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.152639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.153874 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.153876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.154813 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.154814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.154878 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.154881 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:21.154930 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 13:42:21.154970 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":43,"thread_start_us":54,"threads_started":1}
I20260924 13:42:21.158648 21203 tablet_service.cc:1467] Processing CreateTablet for tablet c77c03e85b7944cc86ba8cee4714198f (DEFAULT_TABLE table=test-table [id=272fc95d5248454dba2a531be0ed4a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:21.158996 21203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77c03e85b7944cc86ba8cee4714198f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:21.159349 21289 tablet_bootstrap.cc:492] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5: Bootstrap starting.
I20260924 13:42:21.159776 21290 tablet_bootstrap.cc:492] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d: Bootstrap starting.
I20260924 13:42:21.160145 21289 tablet_bootstrap.cc:654] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:21.160404 21290 tablet_bootstrap.cc:654] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:21.160472 21289 log.cc:824] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:21.160730 21290 log.cc:824] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:21.161281 21289 tablet_bootstrap.cc:492] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5: No bootstrap required, opened a new log
I20260924 13:42:21.161350 21289 ts_tablet_manager.cc:1397] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:21.161391 21290 tablet_bootstrap.cc:492] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d: No bootstrap required, opened a new log
I20260924 13:42:21.161463 21290 ts_tablet_manager.cc:1397] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:21.162949 21290 raft_consensus.cc:348] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } }
I20260924 13:42:21.162977 21289 raft_consensus.cc:348] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } }
I20260924 13:42:21.163085 21289 raft_consensus.cc:374] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:21.163100 21290 raft_consensus.cc:374] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:21.163105 21289 raft_consensus.cc:729] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7175f1bfbd34da8b00d9b640774dfe5, State: Initialized, Role: FOLLOWER
I20260924 13:42:21.163148 21290 raft_consensus.cc:729] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0459a14838fb4be99334b36d74622e7d, State: Initialized, Role: FOLLOWER
I20260924 13:42:21.163226 21289 consensus_queue.cc:260] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } }
I20260924 13:42:21.163255 21290 consensus_queue.cc:260] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } }
I20260924 13:42:21.163506 21290 ts_tablet_manager.cc:1428] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:42:21.163499 21129 heartbeater.cc:503] Master 127.19.53.254:36033 was elected leader, sending a full tablet report...
I20260924 13:42:21.163547 20989 heartbeater.cc:503] Master 127.19.53.254:36033 was elected leader, sending a full tablet report...
I20260924 13:42:21.163676 21293 tablet_bootstrap.cc:492] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41: Bootstrap starting.
I20260924 13:42:21.163482 21289 ts_tablet_manager.cc:1428] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:42:21.164479 21293 tablet_bootstrap.cc:654] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:21.164820 21293 log.cc:824] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:21.165390 21293 tablet_bootstrap.cc:492] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41: No bootstrap required, opened a new log
I20260924 13:42:21.165437 21293 ts_tablet_manager.cc:1397] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:21.166996 21293 raft_consensus.cc:348] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } }
I20260924 13:42:21.167114 21293 raft_consensus.cc:374] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:21.167135 21293 raft_consensus.cc:729] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575cd51400124f479060aa6907920c41, State: Initialized, Role: FOLLOWER
I20260924 13:42:21.167269 21293 consensus_queue.cc:260] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } }
I20260924 13:42:21.167510 21293 ts_tablet_manager.cc:1428] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:42:21.167564 21269 heartbeater.cc:503] Master 127.19.53.254:36033 was elected leader, sending a full tablet report...
W20260924 13:42:21.208053 21130 tablet.cc:2369] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:42:21.265057 21294 raft_consensus.cc:482] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:21.265211 21294 raft_consensus.cc:504] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } }
I20260924 13:42:21.265607 21294 leader_election.cc:288] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 575cd51400124f479060aa6907920c41 (127.19.53.195:39299), 0459a14838fb4be99334b36d74622e7d (127.19.53.193:37537)
I20260924 13:42:21.268396 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.265866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:53393 (local address 127.19.53.193:37537)
0924 13:42:21.265928 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.265930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.265941 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:42:21.265971 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.265972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.265983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.266067 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:21.266259 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.266804 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.267730 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.268058 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.268135 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.268143 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.268306 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.268342 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:21.268392 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.265786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:37537 (local address 127.19.53.194:53393)
0924 13:42:21.265845 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:21.265857 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:21.265900 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:21.266083 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:21.266084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:21.266086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:21.266185 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.266187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.266887 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.266888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.267640 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.267642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.268133 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.268134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.268228 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.268231 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:21.268286 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 13:42:21.268323 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 13:42:21.268538 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.265786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:39299 (local address 127.19.53.194:56703)
0924 13:42:21.266008 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:21.266017 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:21.266092 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:21.266258 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:21.266258 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:21.266260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:21.266331 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.266333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.266984 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.266985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.267712 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.267714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.268263 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.268264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.268344 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.268347 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:21.268408 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 13:42:21.268449 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":21,"thread_start_us":134,"threads_started":1}
I20260924 13:42:21.268532 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.265868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:56703 (local address 127.19.53.195:39299)
0924 13:42:21.265935 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.265937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.266085 (+ 148us) server_negotiation.cc:407] Connection header received
0924 13:42:21.266167 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.266168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.266179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.266241 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:21.266406 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.266898 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.267798 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.268173 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.268264 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.268272 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.268426 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.268467 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 13:42:21.268953 21223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77c03e85b7944cc86ba8cee4714198f" candidate_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575cd51400124f479060aa6907920c41" is_pre_election: true
I20260924 13:42:21.268952 20943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77c03e85b7944cc86ba8cee4714198f" candidate_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0459a14838fb4be99334b36d74622e7d" is_pre_election: true
I20260924 13:42:21.269121 20943 raft_consensus.cc:2454] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7175f1bfbd34da8b00d9b640774dfe5 in term 0.
I20260924 13:42:21.269120 21223 raft_consensus.cc:2454] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7175f1bfbd34da8b00d9b640774dfe5 in term 0.
I20260924 13:42:21.269395 21016 leader_election.cc:302] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575cd51400124f479060aa6907920c41, c7175f1bfbd34da8b00d9b640774dfe5; no voters:
I20260924 13:42:21.269536 21294 raft_consensus.cc:2781] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:21.269636 21294 raft_consensus.cc:482] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:21.269675 21294 raft_consensus.cc:3037] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:21.270637 21294 raft_consensus.cc:504] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } }
I20260924 13:42:21.270761 21294 leader_election.cc:288] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [CANDIDATE]: Term 1 election: Requested vote from peers 575cd51400124f479060aa6907920c41 (127.19.53.195:39299), 0459a14838fb4be99334b36d74622e7d (127.19.53.193:37537)
I20260924 13:42:21.270906 21223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77c03e85b7944cc86ba8cee4714198f" candidate_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575cd51400124f479060aa6907920c41"
I20260924 13:42:21.270996 20943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77c03e85b7944cc86ba8cee4714198f" candidate_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0459a14838fb4be99334b36d74622e7d"
I20260924 13:42:21.270989 21223 raft_consensus.cc:3037] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:21.271099 20943 raft_consensus.cc:3037] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:21.271921 21223 raft_consensus.cc:2454] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7175f1bfbd34da8b00d9b640774dfe5 in term 1.
I20260924 13:42:21.272085 20943 raft_consensus.cc:2454] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7175f1bfbd34da8b00d9b640774dfe5 in term 1.
I20260924 13:42:21.272262 21015 leader_election.cc:302] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0459a14838fb4be99334b36d74622e7d, c7175f1bfbd34da8b00d9b640774dfe5; no voters:
I20260924 13:42:21.272358 21294 raft_consensus.cc:2781] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:21.272580 21294 raft_consensus.cc:686] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [term 1 LEADER]: Becoming Leader. State: Replica: c7175f1bfbd34da8b00d9b640774dfe5, State: Running, Role: LEADER
I20260924 13:42:21.272670 21294 consensus_queue.cc:237] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } }
I20260924 13:42:21.273479 20786 catalog_manager.cc:5495] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7175f1bfbd34da8b00d9b640774dfe5 (127.19.53.194). New cstate: current_term: 1 leader_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } health_report { overall_health: HEALTHY } } }
I20260924 13:42:21.299345 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.296631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.19.53.194:43981)
0924 13:42:21.296678 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.296681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.296739 (+ 58us) server_negotiation.cc:407] Connection header received
0924 13:42:21.296837 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.296838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.296856 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.296930 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:21.297105 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.297633 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.298531 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.298733 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.298802 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.299004 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:21.299100 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:21.299182 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.299225 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.299245 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:42:21.301090 21223 raft_consensus.cc:1261] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41 [term 1 FOLLOWER]: Refusing update from remote peer c7175f1bfbd34da8b00d9b640774dfe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:21.301098 20943 raft_consensus.cc:1261] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d [term 1 FOLLOWER]: Refusing update from remote peer c7175f1bfbd34da8b00d9b640774dfe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:21.301424 21299 consensus_queue.cc:1035] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:21.301563 21294 consensus_queue.cc:1035] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:21.304587 21304 mvcc.cc:205] Tried to move back new op lower bound from 7332894069966430208 to 7332894069853335552. Current Snapshot: MvccSnapshot[applied={T|T < 7332894069966430208}]
I20260924 13:42:21.307579 20786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:21.307672 20786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:21.308410 20786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1: Sending DeleteTablet for 3 replicas of tablet c77c03e85b7944cc86ba8cee4714198f
I20260924 13:42:21.308784 21063 tablet_service.cc:1514] Processing DeleteTablet for tablet c77c03e85b7944cc86ba8cee4714198f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:21 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:57474
I20260924 13:42:21.308784 20923 tablet_service.cc:1514] Processing DeleteTablet for tablet c77c03e85b7944cc86ba8cee4714198f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:21 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:50286
I20260924 13:42:21.308826 21203 tablet_service.cc:1514] Processing DeleteTablet for tablet c77c03e85b7944cc86ba8cee4714198f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:21 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:50300
I20260924 13:42:21.309032 21311 tablet_replica.cc:330] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5: stopping tablet replica
I20260924 13:42:21.309036 20786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 13:42:21.309130 21311 raft_consensus.cc:2229] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [term 1 LEADER]: Raft consensus shutting down.
W20260924 13:42:21.309185 20786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:21.309233 21311 raft_consensus.cc:2258] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:21.309679 21311 ts_tablet_manager.cc:1905] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:21.310387 20923 tablet_service.cc:1467] Processing CreateTablet for tablet e791c510377045ca9e3f60e8b8210913 (DEFAULT_TABLE table=test-table [id=5c84c5c9c243467f929750f7457fe9af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:21.310387 21203 tablet_service.cc:1467] Processing CreateTablet for tablet e791c510377045ca9e3f60e8b8210913 (DEFAULT_TABLE table=test-table [id=5c84c5c9c243467f929750f7457fe9af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:21.310499 21203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e791c510377045ca9e3f60e8b8210913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:21.310499 20923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e791c510377045ca9e3f60e8b8210913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:21.311352 21312 tablet_replica.cc:330] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d: stopping tablet replica
I20260924 13:42:21.311439 21312 raft_consensus.cc:2229] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:21.311522 21312 raft_consensus.cc:2258] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:21.311686 21311 ts_tablet_manager.cc:1918] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:21.311744 21311 log.cc:1196] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/c77c03e85b7944cc86ba8cee4714198f
I20260924 13:42:21.311921 21290 tablet_bootstrap.cc:492] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d: Bootstrap starting.
I20260924 13:42:21.312001 21311 ts_tablet_manager.cc:1939] T c77c03e85b7944cc86ba8cee4714198f P c7175f1bfbd34da8b00d9b640774dfe5: Deleting consensus metadata
I20260924 13:42:21.312386 21293 tablet_bootstrap.cc:492] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41: Bootstrap starting.
I20260924 13:42:21.312628 21290 tablet_bootstrap.cc:654] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:21.312770 21293 tablet_bootstrap.cc:654] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:21.313315 20770 catalog_manager.cc:4841] TS c7175f1bfbd34da8b00d9b640774dfe5 (127.19.53.194:43981): tablet c77c03e85b7944cc86ba8cee4714198f (table test-table [id=272fc95d5248454dba2a531be0ed4a82]) successfully deleted
I20260924 13:42:21.313391 21063 tablet_service.cc:1467] Processing CreateTablet for tablet e791c510377045ca9e3f60e8b8210913 (DEFAULT_TABLE table=test-table [id=5c84c5c9c243467f929750f7457fe9af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:21.313503 21063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e791c510377045ca9e3f60e8b8210913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:21.313681 21290 tablet_bootstrap.cc:492] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d: No bootstrap required, opened a new log
I20260924 13:42:21.313740 21290 ts_tablet_manager.cc:1397] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:21.313882 21290 raft_consensus.cc:348] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.313978 21290 raft_consensus.cc:374] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:21.314006 21290 raft_consensus.cc:729] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0459a14838fb4be99334b36d74622e7d, State: Initialized, Role: FOLLOWER
I20260924 13:42:21.314049 21290 consensus_queue.cc:260] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.314134 21290 ts_tablet_manager.cc:1428] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:21.314723 21289 tablet_bootstrap.cc:492] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5: Bootstrap starting.
I20260924 13:42:21.314837 21312 ts_tablet_manager.cc:1905] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:21.314930 21293 tablet_bootstrap.cc:492] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41: No bootstrap required, opened a new log
I20260924 13:42:21.314975 21293 ts_tablet_manager.cc:1397] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 13:42:21.315140 21289 tablet_bootstrap.cc:654] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:21.315115 21293 raft_consensus.cc:348] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.315196 21293 raft_consensus.cc:374] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:21.315222 21293 raft_consensus.cc:729] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575cd51400124f479060aa6907920c41, State: Initialized, Role: FOLLOWER
I20260924 13:42:21.315264 21293 consensus_queue.cc:260] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.315349 21293 ts_tablet_manager.cc:1428] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:21.315784 21289 tablet_bootstrap.cc:492] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5: No bootstrap required, opened a new log
I20260924 13:42:21.315831 21289 ts_tablet_manager.cc:1397] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:21.315950 21289 raft_consensus.cc:348] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.316026 21289 raft_consensus.cc:374] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:21.316058 21289 raft_consensus.cc:729] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7175f1bfbd34da8b00d9b640774dfe5, State: Initialized, Role: FOLLOWER
I20260924 13:42:21.316118 21289 consensus_queue.cc:260] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.316250 21289 ts_tablet_manager.cc:1428] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:21.316308 21312 ts_tablet_manager.cc:1918] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:21.316375 21312 log.cc:1196] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/wal/wals/c77c03e85b7944cc86ba8cee4714198f
I20260924 13:42:21.316622 21312 ts_tablet_manager.cc:1939] T c77c03e85b7944cc86ba8cee4714198f P 0459a14838fb4be99334b36d74622e7d: Deleting consensus metadata
I20260924 13:42:21.316970 20771 catalog_manager.cc:4841] TS 0459a14838fb4be99334b36d74622e7d (127.19.53.193:37537): tablet c77c03e85b7944cc86ba8cee4714198f (table test-table [id=272fc95d5248454dba2a531be0ed4a82]) successfully deleted
I20260924 13:42:21.318005 21313 tablet_replica.cc:330] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41: stopping tablet replica
I20260924 13:42:21.318104 21313 raft_consensus.cc:2229] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:21.318210 21313 raft_consensus.cc:2258] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:21.318560 21313 ts_tablet_manager.cc:1905] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:21.319823 21313 ts_tablet_manager.cc:1918] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:21.319907 21313 log.cc:1196] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/c77c03e85b7944cc86ba8cee4714198f
I20260924 13:42:21.320114 21313 ts_tablet_manager.cc:1939] T c77c03e85b7944cc86ba8cee4714198f P 575cd51400124f479060aa6907920c41: Deleting consensus metadata
I20260924 13:42:21.320463 20772 catalog_manager.cc:4841] TS 575cd51400124f479060aa6907920c41 (127.19.53.195:39299): tablet c77c03e85b7944cc86ba8cee4714198f (table test-table [id=272fc95d5248454dba2a531be0ed4a82]) successfully deleted
W20260924 13:42:21.363335 21270 tablet.cc:2369] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:42:21.462955 21294 raft_consensus.cc:482] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:21.463052 21294 raft_consensus.cc:504] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.463207 21294 leader_election.cc:288] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0459a14838fb4be99334b36d74622e7d (127.19.53.193:37537), 575cd51400124f479060aa6907920c41 (127.19.53.195:39299)
I20260924 13:42:21.463380 20943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e791c510377045ca9e3f60e8b8210913" candidate_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0459a14838fb4be99334b36d74622e7d" is_pre_election: true
I20260924 13:42:21.463423 21223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e791c510377045ca9e3f60e8b8210913" candidate_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575cd51400124f479060aa6907920c41" is_pre_election: true
I20260924 13:42:21.463500 20943 raft_consensus.cc:2454] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7175f1bfbd34da8b00d9b640774dfe5 in term 0.
I20260924 13:42:21.463505 21223 raft_consensus.cc:2454] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7175f1bfbd34da8b00d9b640774dfe5 in term 0.
I20260924 13:42:21.463692 21015 leader_election.cc:302] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0459a14838fb4be99334b36d74622e7d, c7175f1bfbd34da8b00d9b640774dfe5; no voters:
I20260924 13:42:21.463824 21294 raft_consensus.cc:2781] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:21.463871 21294 raft_consensus.cc:482] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:21.463891 21294 raft_consensus.cc:3037] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:21.464578 21294 raft_consensus.cc:504] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.464711 21294 leader_election.cc:288] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [CANDIDATE]: Term 1 election: Requested vote from peers 0459a14838fb4be99334b36d74622e7d (127.19.53.193:37537), 575cd51400124f479060aa6907920c41 (127.19.53.195:39299)
I20260924 13:42:21.464867 20943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e791c510377045ca9e3f60e8b8210913" candidate_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0459a14838fb4be99334b36d74622e7d"
I20260924 13:42:21.464874 21223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e791c510377045ca9e3f60e8b8210913" candidate_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575cd51400124f479060aa6907920c41"
I20260924 13:42:21.464962 20943 raft_consensus.cc:3037] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:21.464959 21223 raft_consensus.cc:3037] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:21.465552 21223 raft_consensus.cc:2454] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7175f1bfbd34da8b00d9b640774dfe5 in term 1.
I20260924 13:42:21.465714 20943 raft_consensus.cc:2454] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7175f1bfbd34da8b00d9b640774dfe5 in term 1.
I20260924 13:42:21.465709 21016 leader_election.cc:302] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575cd51400124f479060aa6907920c41, c7175f1bfbd34da8b00d9b640774dfe5; no voters:
I20260924 13:42:21.465801 21294 raft_consensus.cc:2781] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:21.465853 21294 raft_consensus.cc:686] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [term 1 LEADER]: Becoming Leader. State: Replica: c7175f1bfbd34da8b00d9b640774dfe5, State: Running, Role: LEADER
I20260924 13:42:21.465924 21294 consensus_queue.cc:237] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.466531 20786 catalog_manager.cc:5495] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7175f1bfbd34da8b00d9b640774dfe5 (127.19.53.194). New cstate: current_term: 1 leader_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } health_report { overall_health: UNKNOWN } } }
Sep 24 13:42:21 dist-test-slave-tfm9 krb5kdc[20741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257340, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:21.521782 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.514756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:36033 (local address 127.0.0.1:50226)
0924 13:42:21.514941 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:21.514950 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:21.515043 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:21.515187 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:21.515188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:21.515492 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:21.515542 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.515544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.516301 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.516302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.516835 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.516837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.517154 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.517155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.517194 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.517678 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:21.517683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:21.519092 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:21.520763 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:21.520770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:21.520773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:21.520976 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:21.521198 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:21.521199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:21.521199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:21.521258 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:21.521503 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:21.521510 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:21.521550 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 13:42:21.521602 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 13:42:21.521684 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
I20260924 13:42:21.521826 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.514785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.19.53.254:36033)
0924 13:42:21.514839 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.514871 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.515024 (+ 153us) server_negotiation.cc:407] Connection header received
0924 13:42:21.515113 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.515115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.515124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.515174 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:21.515621 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.516225 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.516913 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.517093 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.517145 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.519170 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:21.519172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:21.519172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:21.519178 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:21.520651 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:21.521072 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:21.521073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:21.521074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:21.521135 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:21.521317 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:21.521317 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:21.521318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:21.521426 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:21.521479 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.521630 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.521729 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":30}
I20260924 13:42:21.523777 20786 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:50226:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 13:42:21.523952 20786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:21.525439 21063 tablet_service.cc:1467] Processing CreateTablet for tablet b1493b21c6654d0c879f572f10136f0b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ef778fa3f8c4ae0b4279f4dcfc3d43f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:42:21.525450 21203 tablet_service.cc:1467] Processing CreateTablet for tablet b1493b21c6654d0c879f572f10136f0b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ef778fa3f8c4ae0b4279f4dcfc3d43f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:42:21.525605 21063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1493b21c6654d0c879f572f10136f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:21.525636 21203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1493b21c6654d0c879f572f10136f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:21.526904 21289 tablet_bootstrap.cc:492] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5: Bootstrap starting.
I20260924 13:42:21.526983 20923 tablet_service.cc:1467] Processing CreateTablet for tablet b1493b21c6654d0c879f572f10136f0b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ef778fa3f8c4ae0b4279f4dcfc3d43f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:42:21.527127 20923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1493b21c6654d0c879f572f10136f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:21.527274 21289 tablet_bootstrap.cc:654] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:21.527810 21289 tablet_bootstrap.cc:492] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5: No bootstrap required, opened a new log
I20260924 13:42:21.527848 21289 ts_tablet_manager.cc:1397] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:21.527958 21289 raft_consensus.cc:348] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.528020 21289 raft_consensus.cc:374] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:21.528035 21289 raft_consensus.cc:729] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7175f1bfbd34da8b00d9b640774dfe5, State: Initialized, Role: FOLLOWER
I20260924 13:42:21.528083 21289 consensus_queue.cc:260] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.528177 21289 ts_tablet_manager.cc:1428] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:21.528203 21294 tablet_replica.cc:439] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } }
I20260924 13:42:21.528282 21294 tablet_replica.cc:442] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:21.528611 21290 tablet_bootstrap.cc:492] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: Bootstrap starting.
I20260924 13:42:21.528622 21293 tablet_bootstrap.cc:492] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41: Bootstrap starting.
I20260924 13:42:21.528996 21290 tablet_bootstrap.cc:654] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:21.529047 21293 tablet_bootstrap.cc:654] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:21.529628 21290 tablet_bootstrap.cc:492] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: No bootstrap required, opened a new log
I20260924 13:42:21.529662 21293 tablet_bootstrap.cc:492] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41: No bootstrap required, opened a new log
I20260924 13:42:21.529683 21290 ts_tablet_manager.cc:1397] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:21.529714 21293 ts_tablet_manager.cc:1397] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:21.529846 21290 raft_consensus.cc:348] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.529912 21290 raft_consensus.cc:374] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:21.529876 21293 raft_consensus.cc:348] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.529963 21290 raft_consensus.cc:729] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0459a14838fb4be99334b36d74622e7d, State: Initialized, Role: FOLLOWER
I20260924 13:42:21.529964 21293 raft_consensus.cc:374] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:21.530004 21293 raft_consensus.cc:729] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575cd51400124f479060aa6907920c41, State: Initialized, Role: FOLLOWER
I20260924 13:42:21.530012 21290 consensus_queue.cc:260] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.530045 21293 consensus_queue.cc:260] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.530123 21290 ts_tablet_manager.cc:1428] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:21.530129 21293 ts_tablet_manager.cc:1428] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:21.530236 21295 tablet_replica.cc:439] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } }
I20260924 13:42:21.530303 21295 tablet_replica.cc:442] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:21.530427 21297 tablet_replica.cc:439] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } }
I20260924 13:42:21.530527 21297 tablet_replica.cc:442] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:21.554333 21295 raft_consensus.cc:482] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:21.554427 21295 raft_consensus.cc:504] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.554827 21295 leader_election.cc:288] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7175f1bfbd34da8b00d9b640774dfe5 (127.19.53.194:43981), 575cd51400124f479060aa6907920c41 (127.19.53.195:39299)
I20260924 13:42:21.557804 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.555074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:44737 (local address 127.19.53.195:39299)
0924 13:42:21.555121 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.555123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.555218 (+ 95us) server_negotiation.cc:407] Connection header received
0924 13:42:21.555293 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.555294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.555302 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.555363 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:21.555512 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.555999 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.557069 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.557424 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.557506 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.557517 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.557692 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.557731 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:42:21.557832 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.555089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:57501 (local address 127.19.53.194:43981)
0924 13:42:21.555138 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.555141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.555206 (+ 65us) server_negotiation.cc:407] Connection header received
0924 13:42:21.555269 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.555270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.555283 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.555344 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:21.555517 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.556004 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.557074 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.557451 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.557550 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.557559 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.557718 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.557761 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 13:42:21.557847 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.555008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:39299 (local address 127.19.53.193:44737)
0924 13:42:21.555143 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:21.555153 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:21.555228 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:21.555366 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:21.555367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:21.555368 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:21.555446 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.555448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.556093 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.556094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.556965 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.556968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.557529 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.557530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.557610 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.557613 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:21.557674 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 13:42:21.557717 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":61,"thread_start_us":59,"threads_started":1}
I20260924 13:42:21.557866 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.554985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:43981 (local address 127.19.53.193:57501)
0924 13:42:21.555130 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:21.555142 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:21.555207 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:21.555348 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:21.555349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:21.555350 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:21.555435 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.555438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.556111 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.556112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.556974 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.556976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.557550 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.557551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.557637 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.557641 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:21.557711 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:42:21.557755 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260924 13:42:21.558241 21083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1493b21c6654d0c879f572f10136f0b" candidate_uuid: "0459a14838fb4be99334b36d74622e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" is_pre_election: true
I20260924 13:42:21.558281 21223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1493b21c6654d0c879f572f10136f0b" candidate_uuid: "0459a14838fb4be99334b36d74622e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575cd51400124f479060aa6907920c41" is_pre_election: true
I20260924 13:42:21.558370 21223 raft_consensus.cc:2454] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0459a14838fb4be99334b36d74622e7d in term 0.
I20260924 13:42:21.558389 21083 raft_consensus.cc:2454] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0459a14838fb4be99334b36d74622e7d in term 0.
I20260924 13:42:21.558588 20874 leader_election.cc:302] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0459a14838fb4be99334b36d74622e7d, c7175f1bfbd34da8b00d9b640774dfe5; no voters:
I20260924 13:42:21.558748 21295 raft_consensus.cc:2781] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:21.558812 21295 raft_consensus.cc:482] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:21.558842 21295 raft_consensus.cc:3037] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:21.559552 21295 raft_consensus.cc:504] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.559734 21295 leader_election.cc:288] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [CANDIDATE]: Term 1 election: Requested vote from peers c7175f1bfbd34da8b00d9b640774dfe5 (127.19.53.194:43981), 575cd51400124f479060aa6907920c41 (127.19.53.195:39299)
I20260924 13:42:21.559886 21083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1493b21c6654d0c879f572f10136f0b" candidate_uuid: "0459a14838fb4be99334b36d74622e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7175f1bfbd34da8b00d9b640774dfe5"
I20260924 13:42:21.559990 21083 raft_consensus.cc:3037] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:21.559954 21223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1493b21c6654d0c879f572f10136f0b" candidate_uuid: "0459a14838fb4be99334b36d74622e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575cd51400124f479060aa6907920c41"
I20260924 13:42:21.560036 21223 raft_consensus.cc:3037] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:21.560619 21223 raft_consensus.cc:2454] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0459a14838fb4be99334b36d74622e7d in term 1.
I20260924 13:42:21.560750 21083 raft_consensus.cc:2454] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0459a14838fb4be99334b36d74622e7d in term 1.
I20260924 13:42:21.560815 20876 leader_election.cc:302] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0459a14838fb4be99334b36d74622e7d, 575cd51400124f479060aa6907920c41; no voters:
I20260924 13:42:21.560914 21295 raft_consensus.cc:2781] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:21.561044 21295 raft_consensus.cc:686] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [term 1 LEADER]: Becoming Leader. State: Replica: 0459a14838fb4be99334b36d74622e7d, State: Running, Role: LEADER
I20260924 13:42:21.561128 21295 consensus_queue.cc:237] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } }
I20260924 13:42:21.561515 21329 tablet_replica.cc:439] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: TxnStatusTablet state changed. Reason: New leader 0459a14838fb4be99334b36d74622e7d. Latest consensus state: current_term: 1 leader_uuid: "0459a14838fb4be99334b36d74622e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } }
I20260924 13:42:21.561656 21329 tablet_replica.cc:442] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:42:21.561856 21331 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 13:42:21.561826 20786 catalog_manager.cc:5495] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0459a14838fb4be99334b36d74622e7d (127.19.53.193). New cstate: current_term: 1 leader_uuid: "0459a14838fb4be99334b36d74622e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } health_report { overall_health: UNKNOWN } } }
I20260924 13:42:21.561992 21331 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260924 13:42:21.562528 20990 tablet.cc:2369] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:42:21.577071 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.574221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:37537 (local address 127.0.0.1:50302)
0924 13:42:21.574303 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:21.574315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:21.574369 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:21.574535 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:21.574536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:21.574537 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:42:21.574584 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.574586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.575485 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.575486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.576091 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.576094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.576568 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.576569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.576623 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.576639 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:42:21.576911 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:42:21.576918 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 13:42:21.576968 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:42:21.577010 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 13:42:21.577091 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.574287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50302 (local address 127.19.53.193:37537)
0924 13:42:21.574333 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.574335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.574354 (+ 19us) server_negotiation.cc:407] Connection header received
0924 13:42:21.574438 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.574439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.574459 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.574524 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:21.574635 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.575380 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.576184 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.576486 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.576565 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.576728 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:21.576844 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:21.576918 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.576980 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.576999 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:42:21.578322 21223 raft_consensus.cc:1261] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41 [term 1 FOLLOWER]: Refusing update from remote peer 0459a14838fb4be99334b36d74622e7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:21.578370 21083 raft_consensus.cc:1261] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5 [term 1 FOLLOWER]: Refusing update from remote peer 0459a14838fb4be99334b36d74622e7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:21.578619 21295 consensus_queue.cc:1035] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:21.578737 21329 consensus_queue.cc:1035] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:21.579331 21297 tablet_replica.cc:439] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41: TxnStatusTablet state changed. Reason: New leader 0459a14838fb4be99334b36d74622e7d. Latest consensus state: current_term: 1 leader_uuid: "0459a14838fb4be99334b36d74622e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } }
I20260924 13:42:21.579445 21297 tablet_replica.cc:442] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:21.579403 21294 tablet_replica.cc:439] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5: TxnStatusTablet state changed. Reason: New leader 0459a14838fb4be99334b36d74622e7d. Latest consensus state: current_term: 1 leader_uuid: "0459a14838fb4be99334b36d74622e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } }
I20260924 13:42:21.579496 21294 tablet_replica.cc:442] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:21.580338 21295 tablet_replica.cc:439] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0459a14838fb4be99334b36d74622e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } }
I20260924 13:42:21.580447 21295 tablet_replica.cc:442] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:42:21.580508 21295 tablet_replica.cc:439] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0459a14838fb4be99334b36d74622e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } }
I20260924 13:42:21.580567 21295 tablet_replica.cc:442] T b1493b21c6654d0c879f572f10136f0b P 0459a14838fb4be99334b36d74622e7d: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:42:21.580581 21297 tablet_replica.cc:439] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0459a14838fb4be99334b36d74622e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } }
I20260924 13:42:21.580654 21297 tablet_replica.cc:442] T b1493b21c6654d0c879f572f10136f0b P 575cd51400124f479060aa6907920c41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:21.581019 21302 mvcc.cc:205] Tried to move back new op lower bound from 7332894071102550016 to 7332894071034912768. Current Snapshot: MvccSnapshot[applied={T|T < 7332894071102550016}]
I20260924 13:42:21.581282 21294 tablet_replica.cc:439] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0459a14838fb4be99334b36d74622e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 } } peers { permanent_uuid: "c7175f1bfbd34da8b00d9b640774dfe5" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43981 } } peers { permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 } } }
I20260924 13:42:21.581368 21294 tablet_replica.cc:442] T b1493b21c6654d0c879f572f10136f0b P c7175f1bfbd34da8b00d9b640774dfe5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:21.586933 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.584207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:50481 (local address 127.19.53.193:37537)
0924 13:42:21.584243 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.584245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.584362 (+ 117us) server_negotiation.cc:407] Connection header received
0924 13:42:21.584402 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.584402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.584412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.584474 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:21.584572 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.585282 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.586216 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.586504 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.586583 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.586716 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:21.586747 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:21.586799 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.586853 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.586869 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:21.586947 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.584144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:37537 (local address 127.19.53.194:50481)
0924 13:42:21.584286 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:21.584296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:21.584360 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:21.584478 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:21.584479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:21.584480 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:42:21.584518 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.584520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.585380 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.585381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.586135 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.586138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.586583 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.586584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.586628 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.586642 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:42:21.586805 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:42:21.586807 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:42:21.586842 (+ 35us) client_negotiation.cc:240] Negotiation successful
0924 13:42:21.586882 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260924 13:42:21.589066 21223 raft_consensus.cc:1261] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41 [term 1 FOLLOWER]: Refusing update from remote peer c7175f1bfbd34da8b00d9b640774dfe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:21.589217 20943 raft_consensus.cc:1261] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d [term 1 FOLLOWER]: Refusing update from remote peer c7175f1bfbd34da8b00d9b640774dfe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:21.589411 21299 consensus_queue.cc:1035] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "575cd51400124f479060aa6907920c41" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:21.589495 21294 consensus_queue.cc:1035] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0459a14838fb4be99334b36d74622e7d" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:21.591054 21303 mvcc.cc:205] Tried to move back new op lower bound from 7332894071147405312 to 7332894070644871168. Current Snapshot: MvccSnapshot[applied={T|T < 7332894071147405312}]
I20260924 13:42:21.597949 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.595316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:37189 (local address 127.19.53.194:43981)
0924 13:42:21.595387 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.595389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.595398 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:21.595420 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.595421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.595430 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.595505 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:21.595682 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.596211 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.597126 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.597346 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.597443 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.597653 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:21.597688 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:21.597749 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.597848 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.597870 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 13:42:21.598578 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.595134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:43981 (local address 127.19.53.193:37189)
0924 13:42:21.595313 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:21.595324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:21.595374 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:21.595509 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:21.595510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:21.595511 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:42:21.595579 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.595580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.596310 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.596311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.597020 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:21.597023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.597447 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.597449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:21.597519 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.597535 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:42:21.597770 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:42:21.597772 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:42:21.597825 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 13:42:21.597877 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260924 13:42:21.607867 20784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:21.607959 20784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:21.608604 20784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ad4813495024e9b9c4233c652f1e5c1: Sending DeleteTablet for 3 replicas of tablet e791c510377045ca9e3f60e8b8210913
I20260924 13:42:21.608865 21203 tablet_service.cc:1514] Processing DeleteTablet for tablet e791c510377045ca9e3f60e8b8210913 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:21 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:50300
I20260924 13:42:21.608867 20923 tablet_service.cc:1514] Processing DeleteTablet for tablet e791c510377045ca9e3f60e8b8210913 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:21 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:50286
I20260924 13:42:21.608973 21063 tablet_service.cc:1514] Processing DeleteTablet for tablet e791c510377045ca9e3f60e8b8210913 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:21 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:57474
I20260924 13:42:21.609010 21312 tablet_replica.cc:330] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d: stopping tablet replica
I20260924 13:42:21.609079 21313 tablet_replica.cc:330] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41: stopping tablet replica
I20260924 13:42:21.609082 21312 raft_consensus.cc:2229] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:21.609126 21312 raft_consensus.cc:2258] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:21.609133 21313 raft_consensus.cc:2229] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:21.609167 21313 raft_consensus.cc:2258] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:21.609238 21311 tablet_replica.cc:330] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5: stopping tablet replica
I20260924 13:42:21.609295 21311 raft_consensus.cc:2229] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:42:21.609350 21311 raft_consensus.cc:2258] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:21.609612 21312 ts_tablet_manager.cc:1905] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:21.609818 21313 ts_tablet_manager.cc:1905] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:21.611272 21312 ts_tablet_manager.cc:1918] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:21.611361 21312 log.cc:1196] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-0/wal/wals/e791c510377045ca9e3f60e8b8210913
I20260924 13:42:21.611406 21313 ts_tablet_manager.cc:1918] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:21.611465 21313 log.cc:1196] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/e791c510377045ca9e3f60e8b8210913
I20260924 13:42:21.611624 21312 ts_tablet_manager.cc:1939] T e791c510377045ca9e3f60e8b8210913 P 0459a14838fb4be99334b36d74622e7d: Deleting consensus metadata
I20260924 13:42:21.611690 21313 ts_tablet_manager.cc:1939] T e791c510377045ca9e3f60e8b8210913 P 575cd51400124f479060aa6907920c41: Deleting consensus metadata
I20260924 13:42:21.611917 21311 ts_tablet_manager.cc:1905] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:21.611939 20771 catalog_manager.cc:4841] TS 0459a14838fb4be99334b36d74622e7d (127.19.53.193:37537): tablet e791c510377045ca9e3f60e8b8210913 (table test-table [id=5c84c5c9c243467f929750f7457fe9af]) successfully deleted
I20260924 13:42:21.613369 21311 ts_tablet_manager.cc:1918] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:21.613428 21311 log.cc:1196] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/e791c510377045ca9e3f60e8b8210913
I20260924 13:42:21.613633 21311 ts_tablet_manager.cc:1939] T e791c510377045ca9e3f60e8b8210913 P c7175f1bfbd34da8b00d9b640774dfe5: Deleting consensus metadata
I20260924 13:42:21.613832 20772 catalog_manager.cc:4841] TS 575cd51400124f479060aa6907920c41 (127.19.53.195:39299): tablet e791c510377045ca9e3f60e8b8210913 (table test-table [id=5c84c5c9c243467f929750f7457fe9af]) successfully deleted
I20260924 13:42:21.614701 20770 catalog_manager.cc:4841] TS c7175f1bfbd34da8b00d9b640774dfe5 (127.19.53.194:43981): tablet e791c510377045ca9e3f60e8b8210913 (table test-table [id=5c84c5c9c243467f929750f7457fe9af]) successfully deleted
Sep 24 13:42:21 dist-test-slave-tfm9 krb5kdc[20741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257341, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.53.193@KRBTEST.COM
I20260924 13:42:21.624133 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.614672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.19.53.193:37537)
0924 13:42:21.614714 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.614717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.614931 (+ 214us) server_negotiation.cc:407] Connection header received
0924 13:42:21.614991 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.614992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.615003 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.615060 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:21.615723 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.616462 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.617156 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.617468 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.617543 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.620648 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:21.620657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:21.620657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:21.620663 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:21.622825 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:21.623286 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:21.623289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:21.623290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:21.623366 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:21.623584 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:21.623586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:21.623586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:21.623738 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:21.623901 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.623948 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.624022 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 13:42:21.624503 20983 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50308
I20260924 13:42:21.631723 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:21.625356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.19.53.254:36033)
0924 13:42:21.625856 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:21.625858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:21.625868 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:42:21.625888 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:21.625889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:21.625899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:21.626004 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:21.626451 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.627220 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.627858 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:21.628148 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:21.628221 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:21.629249 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:21.629250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:21.629251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:21.629258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:21.630717 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:21.631016 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:21.631017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:21.631017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:21.631077 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:21.631253 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:21.631254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:21.631255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:21.631382 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:21.631470 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:21.631557 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 13:42:21.631628 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475}
W20260924 13:42:21.631892 20784 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50234
I20260924 13:42:21.632484 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 20858
I20260924 13:42:21.638128 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 20998
I20260924 13:42:21.644098 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 21138
I20260924 13:42:21.648537 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 20754
2026-09-24T13:42:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2303 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:21 dist-test-slave-tfm9 krb5kdc[21352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:21 dist-test-slave-tfm9 krb5kdc[21352](info): set up 2 sockets
Sep 24 13:42:21 dist-test-slave-tfm9 krb5kdc[21352](info): commencing operation
krb5kdc: starting...
W20260924 13:42:22.683095 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:22 dist-test-slave-tfm9 krb5kdc[21352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:42:22.799957 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:34059
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:34059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:22.880842 21365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:22.881093 21365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:22.881131 21365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:22.882323 21365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:22.882387 21365 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 13:42:22.882412 21365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:22.882436 21365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:22.882471 21365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:22.882504 21365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:22.884006 21365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:34059
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:34059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:22.884281 21365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:22.884524 21365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:22.889530 21373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:22.889672 21365 server_base.cc:1034] running on GCE node
W20260924 13:42:22.889499 21370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:22.889504 21371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:22.890053 21365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:22.890341 21365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:22.891690 21365 hybrid_clock.cc:648] HybridClock initialized: now 1790257342891675 us; error 31 us; skew 500 ppm
Sep 24 13:42:22 dist-test-slave-tfm9 krb5kdc[21352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257342, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:22.894335 21365 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:42:22.894786 21365 webserver.cc:460] Webserver started at http://127.19.53.254:44145/ using document root <none> and password file <none>
I20260924 13:42:22.895010 21365 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:22.895087 21365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:22.895237 21365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:22.896328 21365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "9207f6062fbe48b88bc200db53b9794b"
format_stamp: "Formatted at 2026-09-24 13:42:22 on dist-test-slave-tfm9"
server_key: "c75821433470e50a26c5dbe197a6c360"
server_key_iv: "fafa39a585df2865a4690c6b382fdd9f"
server_key_version: "encryptionkey@0"
I20260924 13:42:22.896694 21365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "9207f6062fbe48b88bc200db53b9794b"
format_stamp: "Formatted at 2026-09-24 13:42:22 on dist-test-slave-tfm9"
server_key: "c75821433470e50a26c5dbe197a6c360"
server_key_iv: "fafa39a585df2865a4690c6b382fdd9f"
server_key_version: "encryptionkey@0"
I20260924 13:42:22.898121 21365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:42:22.898974 21380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:22.899264 21365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:22.899372 21365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "9207f6062fbe48b88bc200db53b9794b"
format_stamp: "Formatted at 2026-09-24 13:42:22 on dist-test-slave-tfm9"
server_key: "c75821433470e50a26c5dbe197a6c360"
server_key_iv: "fafa39a585df2865a4690c6b382fdd9f"
server_key_version: "encryptionkey@0"
I20260924 13:42:22.899452 21365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:22.912537 21365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:22.914512 21365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:22.914665 21365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:22.918977 21365 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:34059
I20260924 13:42:22.919083 21442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:34059 every 8 connection(s)
I20260924 13:42:22.919379 21365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:22.920032 21443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:22.922317 21443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b: Bootstrap starting.
I20260924 13:42:22.922935 21443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:22.923230 21443 log.cc:824] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:22.923879 21443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b: No bootstrap required, opened a new log
I20260924 13:42:22.924461 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 21365
I20260924 13:42:22.924535 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:22.924714 19671 external_mini_cluster.cc:1428] Setting key ed720b691e5acf200ceff1cbbd8ce94a
I20260924 13:42:22.926460 21443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9207f6062fbe48b88bc200db53b9794b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34059 } }
I20260924 13:42:22.926616 21443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:22.926663 21443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9207f6062fbe48b88bc200db53b9794b, State: Initialized, Role: FOLLOWER
I20260924 13:42:22.926807 21443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9207f6062fbe48b88bc200db53b9794b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34059 } }
I20260924 13:42:22.926895 21443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:22.926939 21443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:22.926990 21443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:22.927534 21443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9207f6062fbe48b88bc200db53b9794b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34059 } }
I20260924 13:42:22.927654 21443 leader_election.cc:302] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9207f6062fbe48b88bc200db53b9794b; no voters:
I20260924 13:42:22.927853 21443 leader_election.cc:288] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:22.927888 21448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:22.927991 21448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [term 1 LEADER]: Becoming Leader. State: Replica: 9207f6062fbe48b88bc200db53b9794b, State: Running, Role: LEADER
I20260924 13:42:22.928103 21443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:22.928076 21448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9207f6062fbe48b88bc200db53b9794b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34059 } }
I20260924 13:42:22.928409 21448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9207f6062fbe48b88bc200db53b9794b. Latest consensus state: current_term: 1 leader_uuid: "9207f6062fbe48b88bc200db53b9794b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9207f6062fbe48b88bc200db53b9794b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34059 } } }
I20260924 13:42:22.928493 21448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:22.928417 21450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9207f6062fbe48b88bc200db53b9794b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9207f6062fbe48b88bc200db53b9794b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34059 } } }
I20260924 13:42:22.928637 21450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:22.928694 21453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:22.929225 21453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:22.930653 21465 mvcc.cc:205] Tried to move back new op lower bound from 7332894076642430976 to 7332894076633944064. Current Snapshot: MvccSnapshot[applied={T|T < 7332894076642430976}]
Sep 24 13:42:22 dist-test-slave-tfm9 krb5kdc[21352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:22.937160 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:22.925250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.19.53.254:34059)
0924 13:42:22.925402 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:22.925410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:22.925438 (+ 28us) server_negotiation.cc:407] Connection header received
0924 13:42:22.926449 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:22.926456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:22.926545 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:22.926710 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:22.927421 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:22.928651 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:22.929844 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:22.930470 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:22.930620 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:22.933266 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:22.933270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:22.933270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:22.933282 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:22.935567 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:22.935892 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:22.935895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:22.935896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:22.936029 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:22.936235 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:22.936236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:22.936236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:22.936571 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:22.936740 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:22.936817 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 13:42:22.936958 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260924 13:42:22.937373 21453 catalog_manager.cc:1347] Generated new cluster ID: 72bbe9cc27284957b62ce294e44e62e6
I20260924 13:42:22.937476 21453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:22.948320 21453 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:22.948813 21453 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:22.955041 21453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b: Generated new TSK 0
I20260924 13:42:22.955242 21453 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:42:22.980906 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34059
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:23.061857 21469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:23.062108 21469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:23.062150 21469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:23.063302 21469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:23.063395 21469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:23.063442 21469 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 13:42:23.063489 21469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:42:23.063525 21469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:23.064991 21469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.53.254:34059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:23.065258 21469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:23.065507 21469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:23.070503 21474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:23.070513 21477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:23.070559 21475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:23.070942 21469 server_base.cc:1034] running on GCE node
I20260924 13:42:23.071167 21469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:23.071442 21469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:23.072664 21469 hybrid_clock.cc:648] HybridClock initialized: now 1790257343072651 us; error 41 us; skew 500 ppm
Sep 24 13:42:23 dist-test-slave-tfm9 krb5kdc[21352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257343, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:23.075227 21469 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:42:23.075686 21469 webserver.cc:460] Webserver started at http://127.19.53.193:46735/ using document root <none> and password file <none>
I20260924 13:42:23.075917 21469 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:23.075981 21469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:23.076123 21469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:23.077114 21469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "11f630c14fc440d689fb5e5f6ca351e5"
format_stamp: "Formatted at 2026-09-24 13:42:23 on dist-test-slave-tfm9"
server_key: "c377b7145deb8f79f719060f1f4446dc"
server_key_iv: "4f1d0b6c6baa49ff3716038894cf40b5"
server_key_version: "encryptionkey@0"
I20260924 13:42:23.077472 21469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "11f630c14fc440d689fb5e5f6ca351e5"
format_stamp: "Formatted at 2026-09-24 13:42:23 on dist-test-slave-tfm9"
server_key: "c377b7145deb8f79f719060f1f4446dc"
server_key_iv: "4f1d0b6c6baa49ff3716038894cf40b5"
server_key_version: "encryptionkey@0"
I20260924 13:42:23.078778 21469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:23.079433 21484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:23.079598 21469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:23.079691 21469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "11f630c14fc440d689fb5e5f6ca351e5"
format_stamp: "Formatted at 2026-09-24 13:42:23 on dist-test-slave-tfm9"
server_key: "c377b7145deb8f79f719060f1f4446dc"
server_key_iv: "4f1d0b6c6baa49ff3716038894cf40b5"
server_key_version: "encryptionkey@0"
I20260924 13:42:23.079764 21469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:23.091575 21469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:23.093660 21469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:23.093808 21469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:23.094347 21469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:23.094398 21469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:23.094434 21469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:23.094456 21469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:42:23 dist-test-slave-tfm9 krb5kdc[21352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257343, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:23.101799 21469 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:44363
I20260924 13:42:23.101969 21599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:44363 every 8 connection(s)
I20260924 13:42:23.102216 21469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:42:23.103642 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.094700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:57063 (local address 127.19.53.254:34059)
0924 13:42:23.094765 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:23.094767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:23.094996 (+ 229us) server_negotiation.cc:407] Connection header received
0924 13:42:23.095572 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:23.095573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:23.095590 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:23.095646 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:23.096585 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.097427 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.098494 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.098708 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.098777 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.100795 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:23.100796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:23.100797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:23.100804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:23.102324 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.102770 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.102771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.102772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.102828 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.103109 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.103110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.103110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.103230 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:23.103291 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:23.103498 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 13:42:23.103577 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:42:23.103868 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.094681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34059 (local address 127.19.53.193:57063)
0924 13:42:23.094870 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:23.094907 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:23.095431 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:23.095760 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:23.095766 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:23.096165 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:23.096465 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.096472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.097524 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.097526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.098396 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.098399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.098808 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.098809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.098916 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.099374 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:23.099381 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:23.100702 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:23.102411 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.102417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.102419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.102687 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.102889 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.102890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.102891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.103037 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.103290 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:23.103295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:23.103408 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 13:42:23.103485 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 13:42:23.103628 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":109,"threads_started":1}
I20260924 13:42:23.104337 21600 heartbeater.cc:347] Connected to a master server at 127.19.53.254:34059
I20260924 13:42:23.104414 21600 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:23.104593 21600 heartbeater.cc:511] Master 127.19.53.254:34059 requested a full tablet report, sending...
I20260924 13:42:23.105085 21396 ts_manager.cc:194] Registered new tserver with Master: 11f630c14fc440d689fb5e5f6ca351e5 (127.19.53.193:44363)
I20260924 13:42:23.105578 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 21469
I20260924 13:42:23.105659 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:42:23.105816 19671 external_mini_cluster.cc:1428] Setting key e95d9d3e77c1a553dd332c25356e6cf6
I20260924 13:42:23.106021 21396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:57063
I20260924 13:42:23.110910 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.105327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34059 (local address 127.19.53.193:46495)
0924 13:42:23.105443 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:23.105453 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:23.105508 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:23.105605 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:23.105605 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:23.105794 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:23.105839 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.105855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.106772 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.106773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.107265 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.107267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.107735 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.107736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.107818 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.108167 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:23.108173 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:23.108549 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:23.109934 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.109935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.109935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.110108 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.110310 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.110311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.110311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.110366 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.110603 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:23.110604 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:23.110647 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 13:42:23.110727 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:42:23.110820 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20260924 13:42:23.110906 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.105386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:46495 (local address 127.19.53.254:34059)
0924 13:42:23.105426 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:23.105428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:23.105493 (+ 65us) server_negotiation.cc:407] Connection header received
0924 13:42:23.105552 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:23.105553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:23.105561 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:23.105603 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:23.105990 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.106675 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.107341 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.107650 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.107724 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.108630 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:23.108632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:23.108632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:23.108637 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:23.109837 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.110190 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.110192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.110192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.110244 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.110429 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.110430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.110431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.110534 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:23.110610 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:23.110750 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 13:42:23.110819 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":19}
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:42:23.130795 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34059
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:23.210949 21609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:23.211174 21609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:23.211198 21609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:23.212349 21609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:23.212390 21609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:23.212410 21609 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 13:42:23.212455 21609 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:42:23.212481 21609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:23.213927 21609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.53.254:34059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:23.214133 21609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:23.214371 21609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:23.219309 21614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:23.219314 21615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:23.219323 21617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:23.219340 21609 server_base.cc:1034] running on GCE node
I20260924 13:42:23.219779 21609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:23.219972 21609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:23.221155 21609 hybrid_clock.cc:648] HybridClock initialized: now 1790257343221137 us; error 33 us; skew 500 ppm
Sep 24 13:42:23 dist-test-slave-tfm9 krb5kdc[21352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257343, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:23.223783 21609 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:42:23.224213 21609 webserver.cc:460] Webserver started at http://127.19.53.194:36381/ using document root <none> and password file <none>
I20260924 13:42:23.224407 21609 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:23.224453 21609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:23.224552 21609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:23.225572 21609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "27c3dbce756e4e65a07fcd29bbd40f0c"
format_stamp: "Formatted at 2026-09-24 13:42:23 on dist-test-slave-tfm9"
server_key: "6c5c4d2a6a5132c003bf87d46b0485c6"
server_key_iv: "e0b71d30d3e88ee308ec070efe4c6f5e"
server_key_version: "encryptionkey@0"
I20260924 13:42:23.225878 21609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "27c3dbce756e4e65a07fcd29bbd40f0c"
format_stamp: "Formatted at 2026-09-24 13:42:23 on dist-test-slave-tfm9"
server_key: "6c5c4d2a6a5132c003bf87d46b0485c6"
server_key_iv: "e0b71d30d3e88ee308ec070efe4c6f5e"
server_key_version: "encryptionkey@0"
I20260924 13:42:23.227274 21609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:23.227922 21624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:23.228232 21609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:23.228320 21609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "27c3dbce756e4e65a07fcd29bbd40f0c"
format_stamp: "Formatted at 2026-09-24 13:42:23 on dist-test-slave-tfm9"
server_key: "6c5c4d2a6a5132c003bf87d46b0485c6"
server_key_iv: "e0b71d30d3e88ee308ec070efe4c6f5e"
server_key_version: "encryptionkey@0"
I20260924 13:42:23.228381 21609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:23.243943 21609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:23.245987 21609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:23.246119 21609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:23.246551 21609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:23.246613 21609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:23.246654 21609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:23.246680 21609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:42:23 dist-test-slave-tfm9 krb5kdc[21352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257343, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:23.253329 21609 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:45157
I20260924 13:42:23.253398 21739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:45157 every 8 connection(s)
I20260924 13:42:23.253736 21609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:42:23.255200 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 21609
I20260924 13:42:23.255307 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:42:23.255520 19671 external_mini_cluster.cc:1428] Setting key 46766700407b18ea2995adfe412eafec
I20260924 13:42:23.256541 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.247071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:38793 (local address 127.19.53.254:34059)
0924 13:42:23.247114 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:23.247117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:23.247365 (+ 248us) server_negotiation.cc:407] Connection header received
0924 13:42:23.247901 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:23.247902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:23.247915 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:23.247967 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:23.248779 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.249621 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.251097 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.251422 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.251499 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.253577 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:23.253578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:23.253579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:23.253587 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:23.254909 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.255524 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.255525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.255525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.255615 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.255871 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.255872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.255872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.255983 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:23.256055 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:23.256374 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 13:42:23.256452 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:42:23.256711 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.247059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34059 (local address 127.19.53.194:38793)
0924 13:42:23.247243 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:23.247292 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:23.247790 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:23.248041 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:23.248045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:23.248400 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:23.248668 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.248675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.249721 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.249722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.251004 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.251006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.251503 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.251503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.251598 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.252067 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:23.252076 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:23.253446 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:23.255037 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.255045 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.255048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.255383 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.255679 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.255680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.255681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.255812 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.256055 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:23.256062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:23.256207 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 13:42:23.256336 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 13:42:23.256486 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260924 13:42:23.257207 21740 heartbeater.cc:347] Connected to a master server at 127.19.53.254:34059
I20260924 13:42:23.257308 21740 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:23.257522 21740 heartbeater.cc:511] Master 127.19.53.254:34059 requested a full tablet report, sending...
I20260924 13:42:23.257886 21397 ts_manager.cc:194] Registered new tserver with Master: 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157)
I20260924 13:42:23.258389 21397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:38793
I20260924 13:42:23.264549 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.258293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34059 (local address 127.19.53.194:35591)
0924 13:42:23.258431 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:23.258442 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:23.258523 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:23.258662 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:23.258662 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:23.258867 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:23.258925 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.258938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.259728 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.259729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.260635 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.260638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.260974 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.260975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.261051 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.261501 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:23.261506 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:23.261794 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:23.263491 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.263492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.263492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.263660 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.263902 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.263903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.263903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.263967 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.264326 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:23.264327 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:23.264361 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 13:42:23.264409 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 13:42:23.264488 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260924 13:42:23.264866 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.258358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:35591 (local address 127.19.53.254:34059)
0924 13:42:23.258392 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:23.258394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:23.258495 (+ 101us) server_negotiation.cc:407] Connection header received
0924 13:42:23.258585 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:23.258586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:23.258597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:23.258655 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:23.259040 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.259617 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.260729 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.260904 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.260954 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.261877 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:23.261878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:23.261879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:23.261886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:23.263358 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.263731 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.263732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.263733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.263790 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.264038 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.264039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.264040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.264250 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:23.264324 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:23.264454 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 13:42:23.264537 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:42:23.280516 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34059
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:23.359850 21749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:23.360041 21749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:23.360067 21749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:23.361120 21749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:23.361153 21749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:23.361172 21749 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 13:42:23.361184 21749 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:42:23.361202 21749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:23.362557 21749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.53.254:34059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:23.362731 21749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:23.362896 21749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:23.367611 21754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:23.367698 21749 server_base.cc:1034] running on GCE node
W20260924 13:42:23.367667 21755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:23.367655 21757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:23.367962 21749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:23.368203 21749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:23.369359 21749 hybrid_clock.cc:648] HybridClock initialized: now 1790257343369352 us; error 33 us; skew 500 ppm
Sep 24 13:42:23 dist-test-slave-tfm9 krb5kdc[21352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257343, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:23.371724 21749 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:42:23.372110 21749 webserver.cc:460] Webserver started at http://127.19.53.195:37051/ using document root <none> and password file <none>
I20260924 13:42:23.372334 21749 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:23.372398 21749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:23.372539 21749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:23.373503 21749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "973e98f67ac548d2bb519dfcad1d7a13"
format_stamp: "Formatted at 2026-09-24 13:42:23 on dist-test-slave-tfm9"
server_key: "394925f57fad7871e9a35c610e7b857f"
server_key_iv: "21a27ba40a4ab8204f9b4d07e0b01716"
server_key_version: "encryptionkey@0"
I20260924 13:42:23.373847 21749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "973e98f67ac548d2bb519dfcad1d7a13"
format_stamp: "Formatted at 2026-09-24 13:42:23 on dist-test-slave-tfm9"
server_key: "394925f57fad7871e9a35c610e7b857f"
server_key_iv: "21a27ba40a4ab8204f9b4d07e0b01716"
server_key_version: "encryptionkey@0"
I20260924 13:42:23.375211 21749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:23.375869 21764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:23.376021 21749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:23.376111 21749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "973e98f67ac548d2bb519dfcad1d7a13"
format_stamp: "Formatted at 2026-09-24 13:42:23 on dist-test-slave-tfm9"
server_key: "394925f57fad7871e9a35c610e7b857f"
server_key_iv: "21a27ba40a4ab8204f9b4d07e0b01716"
server_key_version: "encryptionkey@0"
I20260924 13:42:23.376185 21749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:23.386178 21749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:23.387874 21749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:23.388020 21749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:23.388486 21749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:23.388546 21749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:23.388585 21749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:23.388626 21749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:23.395295 21749 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:33073
I20260924 13:42:23.395354 21879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:33073 every 8 connection(s)
Sep 24 13:42:23 dist-test-slave-tfm9 krb5kdc[21352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257343, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:23.395673 21749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:42:23.398401 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.389019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:38291 (local address 127.19.53.254:34059)
0924 13:42:23.389102 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:23.389105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:23.389256 (+ 151us) server_negotiation.cc:407] Connection header received
0924 13:42:23.389703 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:23.389705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:23.389718 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:23.389822 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:23.390629 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.391515 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.393080 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.393434 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.393517 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.396125 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:23.396126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:23.396127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:23.396135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:23.397279 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.397631 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.397631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.397632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.397681 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.397848 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.397849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.397849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.398001 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:23.398075 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:23.398271 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 13:42:23.398337 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 13:42:23.398545 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.388942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34059 (local address 127.19.53.195:38291)
0924 13:42:23.389122 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:23.389160 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:23.389608 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:23.389854 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:23.389857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:23.390242 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:23.390519 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.390525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.391640 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.391641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.392980 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.392984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.393523 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.393524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.393670 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.394241 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:23.394253 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:23.396000 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:23.397351 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.397356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.397357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.397566 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.397726 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.397726 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.397726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.397804 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.398068 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:23.398071 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:23.398178 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 13:42:23.398258 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:42:23.398370 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260924 13:42:23.399037 21880 heartbeater.cc:347] Connected to a master server at 127.19.53.254:34059
I20260924 13:42:23.399111 21880 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:23.399312 21880 heartbeater.cc:511] Master 127.19.53.254:34059 requested a full tablet report, sending...
I20260924 13:42:23.399659 21397 ts_manager.cc:194] Registered new tserver with Master: 973e98f67ac548d2bb519dfcad1d7a13 (127.19.53.195:33073)
I20260924 13:42:23.400139 21397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:38291
I20260924 13:42:23.404753 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 21749
I20260924 13:42:23.404857 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:42:23.405108 19671 external_mini_cluster.cc:1428] Setting key 13630fdf5587525bc389764b2451af55
I20260924 13:42:23.405324 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.400013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34059 (local address 127.19.53.195:47507)
0924 13:42:23.400145 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:23.400156 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:23.400204 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:23.400381 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:23.400383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:23.400582 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:23.400642 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.400644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.401252 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.401252 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.401835 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.401837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.402220 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.402221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.402273 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.402626 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:23.402630 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:23.402928 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:23.404354 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.404354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.404355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.404510 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.404690 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.404691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.404691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.404758 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.405027 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:23.405027 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:23.405075 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 13:42:23.405148 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 13:42:23.405254 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260924 13:42:23.405371 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.400077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:47507 (local address 127.19.53.254:34059)
0924 13:42:23.400278 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:23.400280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:23.400288 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:23.400306 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:23.400307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:23.400316 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:23.400363 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:23.400719 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.401171 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.401937 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.402144 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.402220 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.403015 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:23.403016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:23.403017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:23.403022 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:23.404280 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.404582 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.404583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.404583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.404630 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.404829 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.404830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.404830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.404979 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:23.405027 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:23.405182 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 13:42:23.405269 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260924 13:42:23.406173 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 13:42:23 dist-test-slave-tfm9 krb5kdc[21352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 13:42:23 dist-test-slave-tfm9 krb5kdc[21352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:23.427789 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.421248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.19.53.254:34059)
0924 13:42:23.421301 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:23.421303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:23.421309 (+ 6us) server_negotiation.cc:407] Connection header received
0924 13:42:23.421332 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:23.421333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:23.421341 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:23.421395 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:23.421965 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.422440 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.423143 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.423315 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.423365 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.425486 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:23.425487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:23.425488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:23.425494 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:23.426813 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.427103 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.427104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.427104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.427163 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.427312 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.427312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.427313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.427405 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:23.427478 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:23.427560 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 13:42:23.427625 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 13:42:23.429399 21397 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:23.431051 21397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:23.436879 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.433111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.19.53.195:33073)
0924 13:42:23.433483 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:23.433488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:23.433497 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:23.433530 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:23.433532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:23.433547 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:23.433642 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:23.433800 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.434526 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.435887 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.436455 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.436547 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.436605 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:23.436755 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 13:42:23.436790 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":85,"threads_started":1}
I20260924 13:42:23.436872 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.432987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:33073 (local address 127.0.0.1:48272)
0924 13:42:23.433270 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:23.433281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:23.433347 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:23.433663 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:23.433663 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:23.433666 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:23.433719 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.433721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.434602 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.434603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.435795 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.435798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.436550 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.436551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.436629 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.436654 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 13:42:23.436739 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 13:42:23.436787 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":51,"threads_started":1}
I20260924 13:42:23.437049 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.433062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.19.53.193:44363)
0924 13:42:23.433229 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:23.433233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:23.433429 (+ 196us) server_negotiation.cc:407] Connection header received
0924 13:42:23.433472 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:23.433474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:23.433490 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:23.433568 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:23.433736 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.434832 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.435842 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.436523 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.436600 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.436653 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:23.436915 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 13:42:23.436950 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260924 13:42:23.437350 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.433164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:45157 (local address 127.0.0.1:52366)
0924 13:42:23.433320 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:23.433328 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:23.433370 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:23.433681 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:23.433682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:23.433684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:23.433721 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.433722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.434930 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.434931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.436007 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.436009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.436926 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.436927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.437181 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.437186 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:42:23.437243 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 13:42:23.437280 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":41,"thread_start_us":40,"threads_started":1}
I20260924 13:42:23.437479 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.432967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:44363 (local address 127.0.0.1:56618)
0924 13:42:23.433232 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:23.433272 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:23.433345 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:23.433587 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:23.433589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:23.433592 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:23.433652 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.433654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.434947 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.434948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.435755 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.435758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.436688 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.436689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.436752 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.436755 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:23.436808 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 13:42:23.436853 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":23,"thread_start_us":47,"threads_started":1}
I20260924 13:42:23.437557 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.433333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52366 (local address 127.19.53.194:45157)
0924 13:42:23.433467 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:23.433471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:23.433479 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:23.433518 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:23.433520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:23.433531 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:23.433621 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:23.433797 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.434834 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.436085 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.436789 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.436847 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.436894 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:23.437432 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 13:42:23.437479 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260924 13:42:23.437693 21814 tablet_service.cc:1467] Processing CreateTablet for tablet ae9315cc291542dd81d56b67219e7632 (DEFAULT_TABLE table=test-table [id=1cbff699abf04d908d990ec2146312d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:23.438040 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9315cc291542dd81d56b67219e7632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:23.438551 21534 tablet_service.cc:1467] Processing CreateTablet for tablet ae9315cc291542dd81d56b67219e7632 (DEFAULT_TABLE table=test-table [id=1cbff699abf04d908d990ec2146312d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:23.438551 21671 tablet_service.cc:1467] Processing CreateTablet for tablet ae9315cc291542dd81d56b67219e7632 (DEFAULT_TABLE table=test-table [id=1cbff699abf04d908d990ec2146312d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:23.438858 21671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9315cc291542dd81d56b67219e7632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:23.438863 21534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9315cc291542dd81d56b67219e7632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:23.441293 21900 tablet_bootstrap.cc:492] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13: Bootstrap starting.
I20260924 13:42:23.442086 21901 tablet_bootstrap.cc:492] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5: Bootstrap starting.
I20260924 13:42:23.442117 21902 tablet_bootstrap.cc:492] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c: Bootstrap starting.
I20260924 13:42:23.442310 21900 tablet_bootstrap.cc:654] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:23.442684 21900 log.cc:824] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:23.442730 21902 tablet_bootstrap.cc:654] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:23.442727 21901 tablet_bootstrap.cc:654] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:23.443022 21902 log.cc:824] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:23.443096 21901 log.cc:824] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:23.443529 21900 tablet_bootstrap.cc:492] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13: No bootstrap required, opened a new log
I20260924 13:42:23.443609 21900 ts_tablet_manager.cc:1397] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:42:23.443784 21902 tablet_bootstrap.cc:492] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c: No bootstrap required, opened a new log
I20260924 13:42:23.443818 21901 tablet_bootstrap.cc:492] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5: No bootstrap required, opened a new log
I20260924 13:42:23.443857 21902 ts_tablet_manager.cc:1397] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:23.443866 21901 ts_tablet_manager.cc:1397] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:23.445261 21900 raft_consensus.cc:348] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.445423 21900 raft_consensus.cc:374] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:23.445467 21900 raft_consensus.cc:729] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973e98f67ac548d2bb519dfcad1d7a13, State: Initialized, Role: FOLLOWER
I20260924 13:42:23.445580 21900 consensus_queue.cc:260] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.445854 21900 ts_tablet_manager.cc:1428] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 13:42:23.445854 21880 heartbeater.cc:503] Master 127.19.53.254:34059 was elected leader, sending a full tablet report...
I20260924 13:42:23.445820 21902 raft_consensus.cc:348] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.445992 21902 raft_consensus.cc:374] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:23.446040 21902 raft_consensus.cc:729] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27c3dbce756e4e65a07fcd29bbd40f0c, State: Initialized, Role: FOLLOWER
I20260924 13:42:23.446060 21901 raft_consensus.cc:348] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.446157 21902 consensus_queue.cc:260] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.446200 21901 raft_consensus.cc:374] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:23.446240 21901 raft_consensus.cc:729] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f630c14fc440d689fb5e5f6ca351e5, State: Initialized, Role: FOLLOWER
I20260924 13:42:23.446357 21901 consensus_queue.cc:260] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.446403 21902 ts_tablet_manager.cc:1428] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:42:23.446419 21740 heartbeater.cc:503] Master 127.19.53.254:34059 was elected leader, sending a full tablet report...
I20260924 13:42:23.446599 21901 ts_tablet_manager.cc:1428] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 13:42:23.446610 21600 heartbeater.cc:503] Master 127.19.53.254:34059 was elected leader, sending a full tablet report...
W20260924 13:42:23.504377 21741 tablet.cc:2369] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:42:23.602908 21601 tablet.cc:2369] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:42:23.646252 21881 tablet.cc:2369] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:42:23.693357 21908 raft_consensus.cc:482] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:23.693487 21908 raft_consensus.cc:504] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.693840 21908 leader_election.cc:288] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157), 973e98f67ac548d2bb519dfcad1d7a13 (127.19.53.195:33073)
I20260924 13:42:23.696727 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.694132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:56113 (local address 127.19.53.194:45157)
0924 13:42:23.694178 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:23.694180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:23.694266 (+ 86us) server_negotiation.cc:407] Connection header received
0924 13:42:23.694349 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:23.694350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:23.694358 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:23.694421 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:23.694572 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.695062 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.695997 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.696388 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.696444 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.696450 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:23.696615 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 13:42:23.696659 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:42:23.696755 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.694049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:45157 (local address 127.19.53.193:56113)
0924 13:42:23.694186 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:23.694197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:23.694264 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:23.694424 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:23.694425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:23.694427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:23.694511 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.694513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.695141 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.695142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.695919 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.695921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.696467 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.696468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.696543 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.696547 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:23.696598 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 13:42:23.696648 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":46,"thread_start_us":66,"threads_started":1}
I20260924 13:42:23.696810 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.694129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:47241 (local address 127.19.53.195:33073)
0924 13:42:23.694190 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:23.694192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:23.694227 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:42:23.694338 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:23.694339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:23.694348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:23.694421 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:23.694632 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.695070 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.696075 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.696460 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.696529 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.696537 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:23.696696 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 13:42:23.696735 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:42:23.696882 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.694026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:33073 (local address 127.19.53.193:47241)
0924 13:42:23.694164 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:23.694177 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:23.694263 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:23.694425 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:23.694426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:23.694428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:23.694530 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.694531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.695160 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.695161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.695988 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.695991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.696540 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.696541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.696624 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.696628 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:23.696683 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 13:42:23.696722 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":26,"thread_start_us":63,"threads_started":1}
I20260924 13:42:23.697214 21834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9315cc291542dd81d56b67219e7632" candidate_uuid: "11f630c14fc440d689fb5e5f6ca351e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973e98f67ac548d2bb519dfcad1d7a13" is_pre_election: true
I20260924 13:42:23.697373 21834 raft_consensus.cc:2454] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f630c14fc440d689fb5e5f6ca351e5 in term 0.
I20260924 13:42:23.697361 21694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9315cc291542dd81d56b67219e7632" candidate_uuid: "11f630c14fc440d689fb5e5f6ca351e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" is_pre_election: true
I20260924 13:42:23.697482 21694 raft_consensus.cc:2454] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f630c14fc440d689fb5e5f6ca351e5 in term 0.
I20260924 13:42:23.697605 21485 leader_election.cc:302] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f630c14fc440d689fb5e5f6ca351e5, 973e98f67ac548d2bb519dfcad1d7a13; no voters:
I20260924 13:42:23.697745 21908 raft_consensus.cc:2781] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:23.697801 21908 raft_consensus.cc:482] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:23.697834 21908 raft_consensus.cc:3037] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:23.698900 21908 raft_consensus.cc:504] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.699060 21908 leader_election.cc:288] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [CANDIDATE]: Term 1 election: Requested vote from peers 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157), 973e98f67ac548d2bb519dfcad1d7a13 (127.19.53.195:33073)
I20260924 13:42:23.699306 21694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9315cc291542dd81d56b67219e7632" candidate_uuid: "11f630c14fc440d689fb5e5f6ca351e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c"
I20260924 13:42:23.699286 21834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9315cc291542dd81d56b67219e7632" candidate_uuid: "11f630c14fc440d689fb5e5f6ca351e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973e98f67ac548d2bb519dfcad1d7a13"
I20260924 13:42:23.699395 21694 raft_consensus.cc:3037] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:23.699397 21834 raft_consensus.cc:3037] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:23.700259 21834 raft_consensus.cc:2454] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f630c14fc440d689fb5e5f6ca351e5 in term 1.
I20260924 13:42:23.700320 21694 raft_consensus.cc:2454] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f630c14fc440d689fb5e5f6ca351e5 in term 1.
I20260924 13:42:23.700491 21485 leader_election.cc:302] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f630c14fc440d689fb5e5f6ca351e5, 973e98f67ac548d2bb519dfcad1d7a13; no voters:
I20260924 13:42:23.700594 21908 raft_consensus.cc:2781] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:23.700695 21908 raft_consensus.cc:686] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 LEADER]: Becoming Leader. State: Replica: 11f630c14fc440d689fb5e5f6ca351e5, State: Running, Role: LEADER
I20260924 13:42:23.700788 21908 consensus_queue.cc:237] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.701807 21396 catalog_manager.cc:5495] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11f630c14fc440d689fb5e5f6ca351e5 (127.19.53.193). New cstate: current_term: 1 leader_uuid: "11f630c14fc440d689fb5e5f6ca351e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } health_report { overall_health: UNKNOWN } } }
Sep 24 13:42:23 dist-test-slave-tfm9 krb5kdc[21352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257342, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:23.744916 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.738529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34059 (local address 127.0.0.1:45694)
0924 13:42:23.738703 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:23.738714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:23.738804 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:23.738941 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:23.738942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:23.739254 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:23.739297 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.739300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.739915 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.739916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.740429 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:23.740431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.740766 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.740767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:23.740823 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.741250 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:23.741254 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:23.742573 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:23.743866 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.743872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.743875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.744055 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.744253 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:23.744254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:23.744254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:23.744311 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:23.744533 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:23.744540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:23.744582 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 13:42:23.744666 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 13:42:23.744772 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260924 13:42:23.744922 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:23.738595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45694 (local address 127.19.53.254:34059)
0924 13:42:23.738683 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:23.738685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:23.738785 (+ 100us) server_negotiation.cc:407] Connection header received
0924 13:42:23.738862 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:23.738862 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:23.738873 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:23.738936 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:23.739377 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.739843 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.740515 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:23.740691 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:23.740765 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:23.742658 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:23.742659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:23.742659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:23.742666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:23.743780 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.744135 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.744136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.744136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.744192 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:23.744369 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:23.744370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:23.744370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:23.744473 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:23.744542 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:23.744687 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 13:42:23.744833 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":61}
I20260924 13:42:23.746778 21396 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:45694:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 13:42:23.746958 21396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:23.748489 21534 tablet_service.cc:1467] Processing CreateTablet for tablet 948b080f75d54cfc99e93859c48efdf6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=726b33b7cbae43c09b23d4ea9b6e1f16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:42:23.748486 21671 tablet_service.cc:1467] Processing CreateTablet for tablet 948b080f75d54cfc99e93859c48efdf6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=726b33b7cbae43c09b23d4ea9b6e1f16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:42:23.748512 21814 tablet_service.cc:1467] Processing CreateTablet for tablet 948b080f75d54cfc99e93859c48efdf6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=726b33b7cbae43c09b23d4ea9b6e1f16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:42:23.748654 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948b080f75d54cfc99e93859c48efdf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:23.748654 21671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948b080f75d54cfc99e93859c48efdf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:23.748664 21534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948b080f75d54cfc99e93859c48efdf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:23.749877 21901 tablet_bootstrap.cc:492] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5: Bootstrap starting.
I20260924 13:42:23.750168 21902 tablet_bootstrap.cc:492] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c: Bootstrap starting.
I20260924 13:42:23.750360 21901 tablet_bootstrap.cc:654] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:23.750954 21902 tablet_bootstrap.cc:654] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:23.751302 21900 tablet_bootstrap.cc:492] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13: Bootstrap starting.
I20260924 13:42:23.751389 21901 tablet_bootstrap.cc:492] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5: No bootstrap required, opened a new log
I20260924 13:42:23.751433 21901 ts_tablet_manager.cc:1397] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:23.751608 21902 tablet_bootstrap.cc:492] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c: No bootstrap required, opened a new log
I20260924 13:42:23.751650 21902 ts_tablet_manager.cc:1397] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:23.751605 21901 raft_consensus.cc:348] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.751695 21901 raft_consensus.cc:374] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:23.751716 21901 raft_consensus.cc:729] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f630c14fc440d689fb5e5f6ca351e5, State: Initialized, Role: FOLLOWER
I20260924 13:42:23.751758 21901 consensus_queue.cc:260] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.751767 21902 raft_consensus.cc:348] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.751814 21902 raft_consensus.cc:374] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:23.751829 21902 raft_consensus.cc:729] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27c3dbce756e4e65a07fcd29bbd40f0c, State: Initialized, Role: FOLLOWER
I20260924 13:42:23.751864 21900 tablet_bootstrap.cc:654] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:23.751863 21901 ts_tablet_manager.cc:1428] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:23.751880 21902 consensus_queue.cc:260] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.751979 21902 ts_tablet_manager.cc:1428] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:23.751992 21908 tablet_replica.cc:439] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } } }
I20260924 13:42:23.752010 21907 tablet_replica.cc:439] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } } }
I20260924 13:42:23.752064 21908 tablet_replica.cc:442] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:23.752081 21907 tablet_replica.cc:442] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:23.752640 21900 tablet_bootstrap.cc:492] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13: No bootstrap required, opened a new log
I20260924 13:42:23.752684 21900 ts_tablet_manager.cc:1397] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:23.752836 21900 raft_consensus.cc:348] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.752892 21900 raft_consensus.cc:374] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:23.752928 21900 raft_consensus.cc:729] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973e98f67ac548d2bb519dfcad1d7a13, State: Initialized, Role: FOLLOWER
I20260924 13:42:23.752972 21900 consensus_queue.cc:260] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:23.753053 21900 ts_tablet_manager.cc:1428] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:23.753134 21906 tablet_replica.cc:439] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } } }
I20260924 13:42:23.753238 21906 tablet_replica.cc:442] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:24.093109 21906 raft_consensus.cc:482] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:24.093225 21906 raft_consensus.cc:504] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:24.093616 21906 leader_election.cc:288] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11f630c14fc440d689fb5e5f6ca351e5 (127.19.53.193:44363), 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157)
I20260924 13:42:24.096568 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:24.094101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:45431 (local address 127.19.53.194:45157)
0924 13:42:24.094147 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:24.094149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:24.094186 (+ 37us) server_negotiation.cc:407] Connection header received
0924 13:42:24.094267 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:24.094268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:24.094277 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:24.094347 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:24.094480 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.094942 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.095855 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.096210 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.096307 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:24.096316 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:24.096461 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 13:42:24.096494 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:42:24.096589 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:24.093880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:49563 (local address 127.19.53.193:44363)
0924 13:42:24.094118 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:24.094120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:24.094128 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:24.094157 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:24.094158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:24.094167 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:24.094256 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:24.094392 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.094881 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.095842 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.096203 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.096295 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:24.096304 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:24.096474 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 13:42:24.096509 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":66,"threads_started":1}
I20260924 13:42:24.096607 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:24.093794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:44363 (local address 127.19.53.195:49563)
0924 13:42:24.093956 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:24.093972 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:24.094056 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:24.094258 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:24.094259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:24.094262 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:24.094321 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:24.094323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.094958 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.094959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:24.095759 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:24.095761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.096294 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.096295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:24.096388 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:24.096391 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:24.096467 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 13:42:24.096509 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":46,"thread_start_us":76,"threads_started":1}
I20260924 13:42:24.096611 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:24.094008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:45157 (local address 127.19.53.195:45431)
0924 13:42:24.094131 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:24.094142 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:24.094202 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:24.094350 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:24.094351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:24.094353 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:24.094410 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:24.094412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.095024 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.095025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:24.095768 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:24.095770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.096292 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.096293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:24.096374 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:24.096378 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:24.096442 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 13:42:24.096486 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260924 13:42:24.097151 21694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948b080f75d54cfc99e93859c48efdf6" candidate_uuid: "973e98f67ac548d2bb519dfcad1d7a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" is_pre_election: true
I20260924 13:42:24.097188 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948b080f75d54cfc99e93859c48efdf6" candidate_uuid: "973e98f67ac548d2bb519dfcad1d7a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f630c14fc440d689fb5e5f6ca351e5" is_pre_election: true
I20260924 13:42:24.097244 21694 raft_consensus.cc:2454] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 973e98f67ac548d2bb519dfcad1d7a13 in term 0.
I20260924 13:42:24.097304 21554 raft_consensus.cc:2454] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 973e98f67ac548d2bb519dfcad1d7a13 in term 0.
I20260924 13:42:24.097430 21767 leader_election.cc:302] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27c3dbce756e4e65a07fcd29bbd40f0c, 973e98f67ac548d2bb519dfcad1d7a13; no voters:
I20260924 13:42:24.097656 21906 raft_consensus.cc:2781] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:24.097704 21906 raft_consensus.cc:482] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:24.097731 21906 raft_consensus.cc:3037] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:24.098469 21906 raft_consensus.cc:504] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:24.098603 21906 leader_election.cc:288] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [CANDIDATE]: Term 1 election: Requested vote from peers 11f630c14fc440d689fb5e5f6ca351e5 (127.19.53.193:44363), 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157)
I20260924 13:42:24.098743 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948b080f75d54cfc99e93859c48efdf6" candidate_uuid: "973e98f67ac548d2bb519dfcad1d7a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f630c14fc440d689fb5e5f6ca351e5"
I20260924 13:42:24.098753 21694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948b080f75d54cfc99e93859c48efdf6" candidate_uuid: "973e98f67ac548d2bb519dfcad1d7a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c"
I20260924 13:42:24.098829 21554 raft_consensus.cc:3037] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:24.098829 21694 raft_consensus.cc:3037] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:24.099475 21694 raft_consensus.cc:2454] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 973e98f67ac548d2bb519dfcad1d7a13 in term 1.
I20260924 13:42:24.099618 21767 leader_election.cc:302] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27c3dbce756e4e65a07fcd29bbd40f0c, 973e98f67ac548d2bb519dfcad1d7a13; no voters:
I20260924 13:42:24.099689 21554 raft_consensus.cc:2454] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 973e98f67ac548d2bb519dfcad1d7a13 in term 1.
I20260924 13:42:24.099695 21906 raft_consensus.cc:2781] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:24.099797 21906 raft_consensus.cc:686] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 LEADER]: Becoming Leader. State: Replica: 973e98f67ac548d2bb519dfcad1d7a13, State: Running, Role: LEADER
I20260924 13:42:24.099881 21906 consensus_queue.cc:237] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:24.100275 21925 tablet_replica.cc:439] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13: TxnStatusTablet state changed. Reason: New leader 973e98f67ac548d2bb519dfcad1d7a13. Latest consensus state: current_term: 1 leader_uuid: "973e98f67ac548d2bb519dfcad1d7a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } } }
I20260924 13:42:24.100356 21925 tablet_replica.cc:442] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:42:24.100589 21927 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 13:42:24.100711 21927 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 13:42:24.100627 21397 catalog_manager.cc:5495] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 973e98f67ac548d2bb519dfcad1d7a13 (127.19.53.195). New cstate: current_term: 1 leader_uuid: "973e98f67ac548d2bb519dfcad1d7a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } health_report { overall_health: HEALTHY } } }
I20260924 13:42:24.157218 21908 consensus_queue.cc:1035] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:24.159341 21911 consensus_queue.cc:1035] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:24.193221 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:24.190445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48284 (local address 127.19.53.195:33073)
0924 13:42:24.190500 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:24.190502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:24.190510 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:24.190604 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:24.190604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:24.190619 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:24.190692 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:24.190839 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.191331 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.192480 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.192658 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.192706 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:24.192902 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:24.193003 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:24.193053 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:24.193132 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 13:42:24.193148 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 13:42:24.193254 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:24.190387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:33073 (local address 127.0.0.1:48284)
0924 13:42:24.190468 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:24.190480 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:24.190525 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:24.190712 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:24.190713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:24.190714 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:42:24.190772 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:24.190775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.191407 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.191408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:24.192402 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:24.192405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.192723 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.192724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:24.192793 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:24.192811 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:42:24.193059 (+ 248us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:42:24.193068 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 13:42:24.193125 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 13:42:24.193176 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 13:42:24.194840 21694 raft_consensus.cc:1261] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Refusing update from remote peer 973e98f67ac548d2bb519dfcad1d7a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:24.194953 21554 raft_consensus.cc:1261] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 FOLLOWER]: Refusing update from remote peer 973e98f67ac548d2bb519dfcad1d7a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:24.195089 21906 consensus_queue.cc:1035] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:24.195205 21925 consensus_queue.cc:1035] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:24.196202 21907 tablet_replica.cc:439] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c: TxnStatusTablet state changed. Reason: New leader 973e98f67ac548d2bb519dfcad1d7a13. Latest consensus state: current_term: 1 leader_uuid: "973e98f67ac548d2bb519dfcad1d7a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } } }
I20260924 13:42:24.196305 21907 tablet_replica.cc:442] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:24.196492 21908 tablet_replica.cc:439] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5: TxnStatusTablet state changed. Reason: New leader 973e98f67ac548d2bb519dfcad1d7a13. Latest consensus state: current_term: 1 leader_uuid: "973e98f67ac548d2bb519dfcad1d7a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } } }
I20260924 13:42:24.196584 21908 tablet_replica.cc:442] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:24.197290 21925 tablet_replica.cc:439] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "973e98f67ac548d2bb519dfcad1d7a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } } }
I20260924 13:42:24.197365 21925 tablet_replica.cc:442] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:42:24.197433 21930 mvcc.cc:205] Tried to move back new op lower bound from 7332894081819869184 to 7332894081433567232. Current Snapshot: MvccSnapshot[applied={T|T < 7332894081819869184}]
I20260924 13:42:24.197461 21925 tablet_replica.cc:439] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "973e98f67ac548d2bb519dfcad1d7a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } } }
I20260924 13:42:24.197497 21925 tablet_replica.cc:442] T 948b080f75d54cfc99e93859c48efdf6 P 973e98f67ac548d2bb519dfcad1d7a13: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:42:24.197521 21908 tablet_replica.cc:439] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "973e98f67ac548d2bb519dfcad1d7a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } } }
I20260924 13:42:24.197576 21908 tablet_replica.cc:442] T 948b080f75d54cfc99e93859c48efdf6 P 11f630c14fc440d689fb5e5f6ca351e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:24.197750 21931 mvcc.cc:205] Tried to move back new op lower bound from 7332894081819869184 to 7332894081433567232. Current Snapshot: MvccSnapshot[applied={T|T < 7332894081819869184}]
I20260924 13:42:24.198549 21907 tablet_replica.cc:439] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "973e98f67ac548d2bb519dfcad1d7a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } } }
I20260924 13:42:24.198643 21907 tablet_replica.cc:442] T 948b080f75d54cfc99e93859c48efdf6 P 27c3dbce756e4e65a07fcd29bbd40f0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:24.199162 21933 mvcc.cc:205] Tried to move back new op lower bound from 7332894081819869184 to 7332894081433567232. Current Snapshot: MvccSnapshot[applied={T|T < 7332894081819869184}]
I20260924 13:42:24.202121 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:24.199596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.19.53.193:44363)
0924 13:42:24.199643 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:24.199645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:24.199653 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:24.199700 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:24.199700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:24.199715 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:24.199778 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:24.199913 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.200333 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.201384 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.201567 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.201614 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:24.201800 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:24.201876 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:24.201969 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:24.202009 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 13:42:24.202039 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:24.206771 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:24.204231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:37827 (local address 127.19.53.195:33073)
0924 13:42:24.204286 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:24.204287 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:24.204339 (+ 52us) server_negotiation.cc:407] Connection header received
0924 13:42:24.204388 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:24.204389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:24.204397 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:24.204457 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:24.204523 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.205162 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.206059 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.206347 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.206422 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:24.206556 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:24.206588 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:24.206652 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:24.206692 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 13:42:24.206710 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:42:24.206779 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:24.204159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:33073 (local address 127.19.53.193:37827)
0924 13:42:24.204286 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:24.204295 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:24.204350 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:24.204459 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:24.204459 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:24.204460 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:42:24.204490 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:24.204491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.205291 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.205292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:24.205979 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:24.205986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.206425 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.206426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:24.206464 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:24.206476 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:42:24.206653 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:42:24.206654 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:42:24.206688 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 13:42:24.206722 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260924 13:42:24.216387 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:24.213859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:59203 (local address 127.19.53.193:44363)
0924 13:42:24.213896 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:24.213898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:24.213954 (+ 56us) server_negotiation.cc:407] Connection header received
0924 13:42:24.214038 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:24.214039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:24.214055 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:24.214100 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:24.214228 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.214930 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.215712 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.215988 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.216055 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:24.216182 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:24.216211 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:24.216259 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:24.216307 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 13:42:24.216326 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:42:24.216403 21948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:24.213718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:44363 (local address 127.19.53.195:59203)
0924 13:42:24.213879 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:24.213890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:24.213969 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:24.214100 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:24.214101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:24.214103 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:42:24.214170 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:24.214171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.215012 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.215013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:24.215635 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:24.215637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:24.216061 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:24.216061 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:24.216105 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:24.216116 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:42:24.216264 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:42:24.216265 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:42:24.216305 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 13:42:24.216337 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260924 13:42:24.223732 21397 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:24.223858 21397 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:24.224519 21397 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b: Sending DeleteTablet for 3 replicas of tablet ae9315cc291542dd81d56b67219e7632
I20260924 13:42:24.224810 21534 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9315cc291542dd81d56b67219e7632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:24 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:56618
I20260924 13:42:24.224974 21814 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9315cc291542dd81d56b67219e7632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:24 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:48272
I20260924 13:42:24.224993 21671 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9315cc291542dd81d56b67219e7632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:24 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:52366
I20260924 13:42:24.225302 21951 tablet_replica.cc:330] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5: stopping tablet replica
I20260924 13:42:24.225332 21397 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:24.225466 21397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:24.225425 21953 tablet_replica.cc:330] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c: stopping tablet replica
I20260924 13:42:24.225756 21953 raft_consensus.cc:2229] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:24.225890 21951 raft_consensus.cc:2229] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:42:24.226027 21953 raft_consensus.cc:2258] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:24.226161 21951 raft_consensus.cc:2258] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:24.226483 21534 tablet_service.cc:1467] Processing CreateTablet for tablet f63d149e0f49488c8ba68b8d3d0967cc (DEFAULT_TABLE table=test-table [id=13456f98032b4d1284d971d6ce6d3ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:24.226621 21534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63d149e0f49488c8ba68b8d3d0967cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:24.226931 21951 ts_tablet_manager.cc:1905] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:24.227141 21953 ts_tablet_manager.cc:1905] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:24.228214 21901 tablet_bootstrap.cc:492] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5: Bootstrap starting.
I20260924 13:42:24.228395 21671 tablet_service.cc:1467] Processing CreateTablet for tablet f63d149e0f49488c8ba68b8d3d0967cc (DEFAULT_TABLE table=test-table [id=13456f98032b4d1284d971d6ce6d3ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:24.228507 21671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63d149e0f49488c8ba68b8d3d0967cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:24.228628 21901 tablet_bootstrap.cc:654] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:24.229611 21814 tablet_service.cc:1467] Processing CreateTablet for tablet f63d149e0f49488c8ba68b8d3d0967cc (DEFAULT_TABLE table=test-table [id=13456f98032b4d1284d971d6ce6d3ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:24.229743 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63d149e0f49488c8ba68b8d3d0967cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:24.230472 21902 tablet_bootstrap.cc:492] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c: Bootstrap starting.
I20260924 13:42:24.230877 21902 tablet_bootstrap.cc:654] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:24.231089 21900 tablet_bootstrap.cc:492] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13: Bootstrap starting.
I20260924 13:42:24.231353 21900 tablet_bootstrap.cc:654] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:24.232057 21953 ts_tablet_manager.cc:1918] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:24.232112 21900 tablet_bootstrap.cc:492] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13: No bootstrap required, opened a new log
I20260924 13:42:24.232165 21900 ts_tablet_manager.cc:1397] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 13:42:24.232277 21900 raft_consensus.cc:348] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:24.232337 21900 raft_consensus.cc:374] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:24.232355 21900 raft_consensus.cc:729] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973e98f67ac548d2bb519dfcad1d7a13, State: Initialized, Role: FOLLOWER
I20260924 13:42:24.232393 21900 consensus_queue.cc:260] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:24.232463 21900 ts_tablet_manager.cc:1428] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:24.232909 21951 ts_tablet_manager.cc:1918] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:24.232992 21952 tablet_replica.cc:330] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13: stopping tablet replica
I20260924 13:42:24.233021 21951 log.cc:1196] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/wal/wals/ae9315cc291542dd81d56b67219e7632
I20260924 13:42:24.233085 21952 raft_consensus.cc:2229] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:24.233405 21951 ts_tablet_manager.cc:1939] T ae9315cc291542dd81d56b67219e7632 P 11f630c14fc440d689fb5e5f6ca351e5: Deleting consensus metadata
I20260924 13:42:24.233511 21953 log.cc:1196] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/ae9315cc291542dd81d56b67219e7632
I20260924 13:42:24.233563 21952 raft_consensus.cc:2258] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:24.233835 21953 ts_tablet_manager.cc:1939] T ae9315cc291542dd81d56b67219e7632 P 27c3dbce756e4e65a07fcd29bbd40f0c: Deleting consensus metadata
I20260924 13:42:24.233661 21901 tablet_bootstrap.cc:492] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5: No bootstrap required, opened a new log
I20260924 13:42:24.233892 21952 ts_tablet_manager.cc:1905] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:24.233978 21901 ts_tablet_manager.cc:1397] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.000s
I20260924 13:42:24.234129 21901 raft_consensus.cc:348] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:24.234192 21901 raft_consensus.cc:374] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:24.234212 21901 raft_consensus.cc:729] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f630c14fc440d689fb5e5f6ca351e5, State: Initialized, Role: FOLLOWER
I20260924 13:42:24.234256 21901 consensus_queue.cc:260] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:24.234346 21901 ts_tablet_manager.cc:1428] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:24.235256 21383 catalog_manager.cc:4841] TS 11f630c14fc440d689fb5e5f6ca351e5 (127.19.53.193:44363): tablet ae9315cc291542dd81d56b67219e7632 (table test-table [id=1cbff699abf04d908d990ec2146312d1]) successfully deleted
I20260924 13:42:24.235657 21952 ts_tablet_manager.cc:1918] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:24.235719 21952 log.cc:1196] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/ae9315cc291542dd81d56b67219e7632
I20260924 13:42:24.235783 21383 catalog_manager.cc:4841] TS 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157): tablet ae9315cc291542dd81d56b67219e7632 (table test-table [id=1cbff699abf04d908d990ec2146312d1]) successfully deleted
I20260924 13:42:24.235948 21952 ts_tablet_manager.cc:1939] T ae9315cc291542dd81d56b67219e7632 P 973e98f67ac548d2bb519dfcad1d7a13: Deleting consensus metadata
I20260924 13:42:24.236277 21902 tablet_bootstrap.cc:492] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c: No bootstrap required, opened a new log
I20260924 13:42:24.236331 21902 ts_tablet_manager.cc:1397] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 13:42:24.236338 21381 catalog_manager.cc:4841] TS 973e98f67ac548d2bb519dfcad1d7a13 (127.19.53.195:33073): tablet ae9315cc291542dd81d56b67219e7632 (table test-table [id=1cbff699abf04d908d990ec2146312d1]) successfully deleted
I20260924 13:42:24.236454 21902 raft_consensus.cc:348] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:24.236514 21902 raft_consensus.cc:374] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:24.236534 21902 raft_consensus.cc:729] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27c3dbce756e4e65a07fcd29bbd40f0c, State: Initialized, Role: FOLLOWER
I20260924 13:42:24.236593 21902 consensus_queue.cc:260] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:24.236676 21902 ts_tablet_manager.cc:1428] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:24.261359 21911 raft_consensus.cc:482] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:24.261425 21911 raft_consensus.cc:504] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:24.261608 21911 leader_election.cc:288] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157), 973e98f67ac548d2bb519dfcad1d7a13 (127.19.53.195:33073)
I20260924 13:42:24.261751 21694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d149e0f49488c8ba68b8d3d0967cc" candidate_uuid: "11f630c14fc440d689fb5e5f6ca351e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" is_pre_election: true
I20260924 13:42:24.261780 21834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d149e0f49488c8ba68b8d3d0967cc" candidate_uuid: "11f630c14fc440d689fb5e5f6ca351e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973e98f67ac548d2bb519dfcad1d7a13" is_pre_election: true
I20260924 13:42:24.261834 21694 raft_consensus.cc:2454] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f630c14fc440d689fb5e5f6ca351e5 in term 0.
I20260924 13:42:24.261845 21834 raft_consensus.cc:2454] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f630c14fc440d689fb5e5f6ca351e5 in term 0.
I20260924 13:42:24.262014 21487 leader_election.cc:302] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f630c14fc440d689fb5e5f6ca351e5, 27c3dbce756e4e65a07fcd29bbd40f0c; no voters:
I20260924 13:42:24.262132 21911 raft_consensus.cc:2781] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:24.262169 21911 raft_consensus.cc:482] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:24.262183 21911 raft_consensus.cc:3037] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:24.262827 21911 raft_consensus.cc:504] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:24.262957 21911 leader_election.cc:288] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [CANDIDATE]: Term 1 election: Requested vote from peers 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157), 973e98f67ac548d2bb519dfcad1d7a13 (127.19.53.195:33073)
I20260924 13:42:24.263072 21694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d149e0f49488c8ba68b8d3d0967cc" candidate_uuid: "11f630c14fc440d689fb5e5f6ca351e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c"
I20260924 13:42:24.263123 21694 raft_consensus.cc:3037] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:24.263091 21834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d149e0f49488c8ba68b8d3d0967cc" candidate_uuid: "11f630c14fc440d689fb5e5f6ca351e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973e98f67ac548d2bb519dfcad1d7a13"
I20260924 13:42:24.263159 21834 raft_consensus.cc:3037] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:24.263644 21694 raft_consensus.cc:2454] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f630c14fc440d689fb5e5f6ca351e5 in term 1.
I20260924 13:42:24.263787 21487 leader_election.cc:302] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f630c14fc440d689fb5e5f6ca351e5, 27c3dbce756e4e65a07fcd29bbd40f0c; no voters:
I20260924 13:42:24.263854 21911 raft_consensus.cc:2781] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:24.263882 21911 raft_consensus.cc:686] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 LEADER]: Becoming Leader. State: Replica: 11f630c14fc440d689fb5e5f6ca351e5, State: Running, Role: LEADER
I20260924 13:42:24.263887 21834 raft_consensus.cc:2454] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f630c14fc440d689fb5e5f6ca351e5 in term 1.
I20260924 13:42:24.263926 21911 consensus_queue.cc:237] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:24.264448 21397 catalog_manager.cc:5495] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11f630c14fc440d689fb5e5f6ca351e5 (127.19.53.193). New cstate: current_term: 1 leader_uuid: "11f630c14fc440d689fb5e5f6ca351e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } health_report { overall_health: UNKNOWN } } }
I20260924 13:42:24.271870 21694 raft_consensus.cc:1261] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Refusing update from remote peer 11f630c14fc440d689fb5e5f6ca351e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:24.272049 21834 raft_consensus.cc:1261] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Refusing update from remote peer 11f630c14fc440d689fb5e5f6ca351e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:24.272142 21911 consensus_queue.cc:1035] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:24.272284 21911 consensus_queue.cc:1035] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 13:42:24.276846 21930 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 13:42:25.797071 21397 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:25.797189 21397 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:25.798091 21397 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b: Sending DeleteTablet for 3 replicas of tablet f63d149e0f49488c8ba68b8d3d0967cc
I20260924 13:42:25.798391 21813 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d149e0f49488c8ba68b8d3d0967cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:25 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:48272
I20260924 13:42:25.798373 21534 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d149e0f49488c8ba68b8d3d0967cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:25 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:56618
I20260924 13:42:25.798410 21671 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d149e0f49488c8ba68b8d3d0967cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:25 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:52366
I20260924 13:42:25.799892 21984 tablet_replica.cc:330] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c: stopping tablet replica
I20260924 13:42:25.800026 21984 raft_consensus.cc:2229] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:25.800889 21984 raft_consensus.cc:2258] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:25.801178 21984 ts_tablet_manager.cc:1905] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:25.802035 21983 tablet_replica.cc:330] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5: stopping tablet replica
I20260924 13:42:25.802116 21983 raft_consensus.cc:2229] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:42:25.802192 21983 raft_consensus.cc:2258] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:25.802313 21982 tablet_replica.cc:330] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13: stopping tablet replica
I20260924 13:42:25.802417 21982 raft_consensus.cc:2229] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:25.802508 21983 ts_tablet_manager.cc:1905] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:25.802542 21982 raft_consensus.cc:2258] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:25.802742 21984 ts_tablet_manager.cc:1918] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 13:42:25.802801 21984 log.cc:1196] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/f63d149e0f49488c8ba68b8d3d0967cc
I20260924 13:42:25.802977 21982 ts_tablet_manager.cc:1905] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:25.803015 21984 ts_tablet_manager.cc:1939] T f63d149e0f49488c8ba68b8d3d0967cc P 27c3dbce756e4e65a07fcd29bbd40f0c: Deleting consensus metadata
I20260924 13:42:25.803433 21383 catalog_manager.cc:4841] TS 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157): tablet f63d149e0f49488c8ba68b8d3d0967cc (table test-table [id=13456f98032b4d1284d971d6ce6d3ae8]) successfully deleted
I20260924 13:42:25.804569 21983 ts_tablet_manager.cc:1918] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 13:42:25.804636 21983 log.cc:1196] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/wal/wals/f63d149e0f49488c8ba68b8d3d0967cc
I20260924 13:42:25.804769 21982 ts_tablet_manager.cc:1918] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 13:42:25.804860 21982 log.cc:1196] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/f63d149e0f49488c8ba68b8d3d0967cc
I20260924 13:42:25.804895 21983 ts_tablet_manager.cc:1939] T f63d149e0f49488c8ba68b8d3d0967cc P 11f630c14fc440d689fb5e5f6ca351e5: Deleting consensus metadata
I20260924 13:42:25.805172 21982 ts_tablet_manager.cc:1939] T f63d149e0f49488c8ba68b8d3d0967cc P 973e98f67ac548d2bb519dfcad1d7a13: Deleting consensus metadata
I20260924 13:42:25.805176 21383 catalog_manager.cc:4841] TS 11f630c14fc440d689fb5e5f6ca351e5 (127.19.53.193:44363): tablet f63d149e0f49488c8ba68b8d3d0967cc (table test-table [id=13456f98032b4d1284d971d6ce6d3ae8]) successfully deleted
I20260924 13:42:25.805508 21381 catalog_manager.cc:4841] TS 973e98f67ac548d2bb519dfcad1d7a13 (127.19.53.195:33073): tablet f63d149e0f49488c8ba68b8d3d0967cc (table test-table [id=13456f98032b4d1284d971d6ce6d3ae8]) successfully deleted
Sep 24 13:42:25 dist-test-slave-tfm9 krb5kdc[21352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 13:42:25 dist-test-slave-tfm9 krb5kdc[21352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:25.834986 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:25.815520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.19.53.254:34059)
0924 13:42:25.815657 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:25.815659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:25.815672 (+ 13us) server_negotiation.cc:407] Connection header received
0924 13:42:25.815747 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:25.815748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:25.815758 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:25.815828 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:25.816425 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:25.816936 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:25.817668 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:25.817868 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:25.818004 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:25.832477 (+ 14473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:25.832479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:25.832479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:25.832493 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:25.833821 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:25.834192 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:25.834194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:25.834194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:25.834251 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:25.834442 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:25.834443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:25.834443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:25.834566 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:25.834672 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:25.834781 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 13:42:25.834869 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":85,"threads_started":1}
I20260924 13:42:25.836364 21397 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:25.836514 21397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:25.837864 21534 tablet_service.cc:1467] Processing CreateTablet for tablet f7de9d0a51a5472da27c414aeb596c27 (DEFAULT_TABLE table=test-table [id=50a618e227b04e28ae20b850eb6516b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:25.837899 21671 tablet_service.cc:1467] Processing CreateTablet for tablet f7de9d0a51a5472da27c414aeb596c27 (DEFAULT_TABLE table=test-table [id=50a618e227b04e28ae20b850eb6516b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:25.838017 21813 tablet_service.cc:1467] Processing CreateTablet for tablet f7de9d0a51a5472da27c414aeb596c27 (DEFAULT_TABLE table=test-table [id=50a618e227b04e28ae20b850eb6516b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:25.838054 21671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7de9d0a51a5472da27c414aeb596c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:25.838053 21534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7de9d0a51a5472da27c414aeb596c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:25.838150 21813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7de9d0a51a5472da27c414aeb596c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:25.839579 21994 tablet_bootstrap.cc:492] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5: Bootstrap starting.
I20260924 13:42:25.839666 21993 tablet_bootstrap.cc:492] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c: Bootstrap starting.
I20260924 13:42:25.839779 21995 tablet_bootstrap.cc:492] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13: Bootstrap starting.
I20260924 13:42:25.839970 21994 tablet_bootstrap.cc:654] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:25.840090 21995 tablet_bootstrap.cc:654] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:25.840087 21993 tablet_bootstrap.cc:654] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:25.840636 21994 tablet_bootstrap.cc:492] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5: No bootstrap required, opened a new log
I20260924 13:42:25.840668 21995 tablet_bootstrap.cc:492] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13: No bootstrap required, opened a new log
I20260924 13:42:25.840706 21994 ts_tablet_manager.cc:1397] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:25.840723 21995 ts_tablet_manager.cc:1397] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:25.840859 21994 raft_consensus.cc:348] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:25.840866 21995 raft_consensus.cc:348] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:25.840920 21993 tablet_bootstrap.cc:492] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c: No bootstrap required, opened a new log
I20260924 13:42:25.840941 21995 raft_consensus.cc:374] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:25.840941 21994 raft_consensus.cc:374] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:25.840976 21994 raft_consensus.cc:729] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f630c14fc440d689fb5e5f6ca351e5, State: Initialized, Role: FOLLOWER
I20260924 13:42:25.840966 21995 raft_consensus.cc:729] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973e98f67ac548d2bb519dfcad1d7a13, State: Initialized, Role: FOLLOWER
I20260924 13:42:25.840981 21993 ts_tablet_manager.cc:1397] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:25.841032 21994 consensus_queue.cc:260] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:25.841145 21994 ts_tablet_manager.cc:1428] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:25.841138 21993 raft_consensus.cc:348] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:25.841217 21993 raft_consensus.cc:374] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:25.841073 21995 consensus_queue.cc:260] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:25.841243 21993 raft_consensus.cc:729] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27c3dbce756e4e65a07fcd29bbd40f0c, State: Initialized, Role: FOLLOWER
I20260924 13:42:25.841315 21993 consensus_queue.cc:260] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:25.841478 21993 ts_tablet_manager.cc:1428] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:25.841904 21995 ts_tablet_manager.cc:1428] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:26.216195 21977 raft_consensus.cc:482] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:26.216323 21977 raft_consensus.cc:504] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:26.216532 21977 leader_election.cc:288] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157), 973e98f67ac548d2bb519dfcad1d7a13 (127.19.53.195:33073)
I20260924 13:42:26.216771 21834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7de9d0a51a5472da27c414aeb596c27" candidate_uuid: "11f630c14fc440d689fb5e5f6ca351e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973e98f67ac548d2bb519dfcad1d7a13" is_pre_election: true
I20260924 13:42:26.216784 21694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7de9d0a51a5472da27c414aeb596c27" candidate_uuid: "11f630c14fc440d689fb5e5f6ca351e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" is_pre_election: true
I20260924 13:42:26.216881 21834 raft_consensus.cc:2454] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f630c14fc440d689fb5e5f6ca351e5 in term 0.
I20260924 13:42:26.216882 21694 raft_consensus.cc:2454] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f630c14fc440d689fb5e5f6ca351e5 in term 0.
I20260924 13:42:26.217051 21487 leader_election.cc:302] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f630c14fc440d689fb5e5f6ca351e5, 27c3dbce756e4e65a07fcd29bbd40f0c; no voters:
I20260924 13:42:26.217165 21977 raft_consensus.cc:2781] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:26.217214 21977 raft_consensus.cc:482] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:26.217245 21977 raft_consensus.cc:3037] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:26.218087 21977 raft_consensus.cc:504] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:26.218221 21977 leader_election.cc:288] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [CANDIDATE]: Term 1 election: Requested vote from peers 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157), 973e98f67ac548d2bb519dfcad1d7a13 (127.19.53.195:33073)
I20260924 13:42:26.218374 21834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7de9d0a51a5472da27c414aeb596c27" candidate_uuid: "11f630c14fc440d689fb5e5f6ca351e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973e98f67ac548d2bb519dfcad1d7a13"
I20260924 13:42:26.218374 21694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7de9d0a51a5472da27c414aeb596c27" candidate_uuid: "11f630c14fc440d689fb5e5f6ca351e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c"
I20260924 13:42:26.218448 21694 raft_consensus.cc:3037] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:26.218448 21834 raft_consensus.cc:3037] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:26.219215 21694 raft_consensus.cc:2454] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f630c14fc440d689fb5e5f6ca351e5 in term 1.
I20260924 13:42:26.219298 21834 raft_consensus.cc:2454] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f630c14fc440d689fb5e5f6ca351e5 in term 1.
I20260924 13:42:26.219389 21487 leader_election.cc:302] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f630c14fc440d689fb5e5f6ca351e5, 27c3dbce756e4e65a07fcd29bbd40f0c; no voters:
I20260924 13:42:26.219473 21977 raft_consensus.cc:2781] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:26.219600 21977 raft_consensus.cc:686] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 LEADER]: Becoming Leader. State: Replica: 11f630c14fc440d689fb5e5f6ca351e5, State: Running, Role: LEADER
I20260924 13:42:26.219659 21977 consensus_queue.cc:237] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:26.220315 21396 catalog_manager.cc:5495] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11f630c14fc440d689fb5e5f6ca351e5 (127.19.53.193). New cstate: current_term: 1 leader_uuid: "11f630c14fc440d689fb5e5f6ca351e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } health_report { overall_health: UNKNOWN } } }
I20260924 13:42:26.283886 22002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:26.281717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.19.53.193:44363)
0924 13:42:26.281858 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:26.281860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:26.281867 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:42:26.281886 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:26.281886 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:26.281895 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:26.281997 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:26.282092 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:26.282586 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:26.283336 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:26.283522 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:26.283595 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:26.283700 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:26.283728 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:26.283767 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:26.283807 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 13:42:26.283826 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":95,"threads_started":1}
I20260924 13:42:26.285974 21694 raft_consensus.cc:1261] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Refusing update from remote peer 11f630c14fc440d689fb5e5f6ca351e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:26.286036 21834 raft_consensus.cc:1261] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Refusing update from remote peer 11f630c14fc440d689fb5e5f6ca351e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:26.286191 21977 consensus_queue.cc:1035] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:26.286280 21973 consensus_queue.cc:1035] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:26.295473 21396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:26.295562 21396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:26.296133 21396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b: Sending DeleteTablet for 3 replicas of tablet f7de9d0a51a5472da27c414aeb596c27
I20260924 13:42:26.296389 21534 tablet_service.cc:1514] Processing DeleteTablet for tablet f7de9d0a51a5472da27c414aeb596c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:26 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:56618
I20260924 13:42:26.296425 21671 tablet_service.cc:1514] Processing DeleteTablet for tablet f7de9d0a51a5472da27c414aeb596c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:26 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:52366
I20260924 13:42:26.296486 21983 tablet_replica.cc:330] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5: stopping tablet replica
I20260924 13:42:26.296506 21984 tablet_replica.cc:330] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c: stopping tablet replica
I20260924 13:42:26.296540 21983 raft_consensus.cc:2229] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:42:26.296577 21984 raft_consensus.cc:2229] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:26.296586 21983 raft_consensus.cc:2258] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:26.296627 21984 raft_consensus.cc:2258] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:26.296713 21396 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:26.296882 21396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:26.296972 21983 ts_tablet_manager.cc:1905] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:26.298125 21534 tablet_service.cc:1467] Processing CreateTablet for tablet 21592b0b643e4358b6777c4bd36ea6ed (DEFAULT_TABLE table=test-table [id=86f10643cdf64e69a39a716f498591f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:26.298143 21814 tablet_service.cc:1467] Processing CreateTablet for tablet 21592b0b643e4358b6777c4bd36ea6ed (DEFAULT_TABLE table=test-table [id=86f10643cdf64e69a39a716f498591f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:26.298228 21534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21592b0b643e4358b6777c4bd36ea6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:26.298264 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21592b0b643e4358b6777c4bd36ea6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:26.298763 21983 ts_tablet_manager.cc:1918] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:26.298753 21671 tablet_service.cc:1467] Processing CreateTablet for tablet 21592b0b643e4358b6777c4bd36ea6ed (DEFAULT_TABLE table=test-table [id=86f10643cdf64e69a39a716f498591f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:26.298841 21983 log.cc:1196] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-0/wal/wals/f7de9d0a51a5472da27c414aeb596c27
I20260924 13:42:26.298866 21671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21592b0b643e4358b6777c4bd36ea6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:26.299079 21983 ts_tablet_manager.cc:1939] T f7de9d0a51a5472da27c414aeb596c27 P 11f630c14fc440d689fb5e5f6ca351e5: Deleting consensus metadata
I20260924 13:42:26.299444 21994 tablet_bootstrap.cc:492] T 21592b0b643e4358b6777c4bd36ea6ed P 11f630c14fc440d689fb5e5f6ca351e5: Bootstrap starting.
I20260924 13:42:26.299682 21995 tablet_bootstrap.cc:492] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13: Bootstrap starting.
I20260924 13:42:26.299743 21994 tablet_bootstrap.cc:654] T 21592b0b643e4358b6777c4bd36ea6ed P 11f630c14fc440d689fb5e5f6ca351e5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:26.300047 21995 tablet_bootstrap.cc:654] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:26.300110 21383 catalog_manager.cc:4841] TS 11f630c14fc440d689fb5e5f6ca351e5 (127.19.53.193:44363): tablet f7de9d0a51a5472da27c414aeb596c27 (table test-table [id=50a618e227b04e28ae20b850eb6516b0]) successfully deleted
I20260924 13:42:26.300208 21993 tablet_bootstrap.cc:492] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c: Bootstrap starting.
I20260924 13:42:26.300439 21994 tablet_bootstrap.cc:492] T 21592b0b643e4358b6777c4bd36ea6ed P 11f630c14fc440d689fb5e5f6ca351e5: No bootstrap required, opened a new log
I20260924 13:42:26.300503 21994 ts_tablet_manager.cc:1397] T 21592b0b643e4358b6777c4bd36ea6ed P 11f630c14fc440d689fb5e5f6ca351e5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:26.300643 21993 tablet_bootstrap.cc:654] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:26.300614 21994 raft_consensus.cc:348] T 21592b0b643e4358b6777c4bd36ea6ed P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:26.300676 21994 raft_consensus.cc:374] T 21592b0b643e4358b6777c4bd36ea6ed P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:26.300683 21995 tablet_bootstrap.cc:492] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13: No bootstrap required, opened a new log
I20260924 13:42:26.300701 21994 raft_consensus.cc:729] T 21592b0b643e4358b6777c4bd36ea6ed P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f630c14fc440d689fb5e5f6ca351e5, State: Initialized, Role: FOLLOWER
I20260924 13:42:26.300724 21995 ts_tablet_manager.cc:1397] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:26.300751 21994 consensus_queue.cc:260] T 21592b0b643e4358b6777c4bd36ea6ed P 11f630c14fc440d689fb5e5f6ca351e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:26.300840 21994 ts_tablet_manager.cc:1428] T 21592b0b643e4358b6777c4bd36ea6ed P 11f630c14fc440d689fb5e5f6ca351e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:26.300817 21995 raft_consensus.cc:348] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:26.300873 21995 raft_consensus.cc:374] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:26.300895 21995 raft_consensus.cc:729] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973e98f67ac548d2bb519dfcad1d7a13, State: Initialized, Role: FOLLOWER
I20260924 13:42:26.300931 21995 consensus_queue.cc:260] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:26.301029 21995 ts_tablet_manager.cc:1428] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:26.301080 21984 ts_tablet_manager.cc:1905] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:26.301977 21813 tablet_service.cc:1514] Processing DeleteTablet for tablet f7de9d0a51a5472da27c414aeb596c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:26 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:48272
I20260924 13:42:26.302109 21982 tablet_replica.cc:330] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13: stopping tablet replica
I20260924 13:42:26.302178 21982 raft_consensus.cc:2229] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:26.302237 21982 raft_consensus.cc:2258] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:26.302294 21993 tablet_bootstrap.cc:492] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c: No bootstrap required, opened a new log
I20260924 13:42:26.302340 21993 ts_tablet_manager.cc:1397] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:26.302479 21993 raft_consensus.cc:348] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:26.302510 21984 ts_tablet_manager.cc:1918] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:26.302562 21993 raft_consensus.cc:374] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:26.302603 21982 ts_tablet_manager.cc:1905] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:26.302605 21993 raft_consensus.cc:729] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27c3dbce756e4e65a07fcd29bbd40f0c, State: Initialized, Role: FOLLOWER
I20260924 13:42:26.302610 21984 log.cc:1196] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/f7de9d0a51a5472da27c414aeb596c27
I20260924 13:42:26.302655 21993 consensus_queue.cc:260] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:26.302778 21993 ts_tablet_manager.cc:1428] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:26.302856 21984 ts_tablet_manager.cc:1939] T f7de9d0a51a5472da27c414aeb596c27 P 27c3dbce756e4e65a07fcd29bbd40f0c: Deleting consensus metadata
I20260924 13:42:26.303921 21383 catalog_manager.cc:4841] TS 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157): tablet f7de9d0a51a5472da27c414aeb596c27 (table test-table [id=50a618e227b04e28ae20b850eb6516b0]) successfully deleted
I20260924 13:42:26.303948 21982 ts_tablet_manager.cc:1918] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:26.303985 21982 log.cc:1196] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/f7de9d0a51a5472da27c414aeb596c27
I20260924 13:42:26.304235 21982 ts_tablet_manager.cc:1939] T f7de9d0a51a5472da27c414aeb596c27 P 973e98f67ac548d2bb519dfcad1d7a13: Deleting consensus metadata
I20260924 13:42:26.304529 21381 catalog_manager.cc:4841] TS 973e98f67ac548d2bb519dfcad1d7a13 (127.19.53.195:33073): tablet f7de9d0a51a5472da27c414aeb596c27 (table test-table [id=50a618e227b04e28ae20b850eb6516b0]) successfully deleted
I20260924 13:42:26.519184 21963 raft_consensus.cc:482] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:26.519304 21963 raft_consensus.cc:504] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:26.519484 21963 leader_election.cc:288] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11f630c14fc440d689fb5e5f6ca351e5 (127.19.53.193:44363), 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157)
I20260924 13:42:26.519729 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21592b0b643e4358b6777c4bd36ea6ed" candidate_uuid: "973e98f67ac548d2bb519dfcad1d7a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f630c14fc440d689fb5e5f6ca351e5" is_pre_election: true
I20260924 13:42:26.519780 21694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21592b0b643e4358b6777c4bd36ea6ed" candidate_uuid: "973e98f67ac548d2bb519dfcad1d7a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" is_pre_election: true
I20260924 13:42:26.519882 21554 raft_consensus.cc:2454] T 21592b0b643e4358b6777c4bd36ea6ed P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 973e98f67ac548d2bb519dfcad1d7a13 in term 0.
I20260924 13:42:26.519891 21694 raft_consensus.cc:2454] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 973e98f67ac548d2bb519dfcad1d7a13 in term 0.
I20260924 13:42:26.520087 21767 leader_election.cc:302] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f630c14fc440d689fb5e5f6ca351e5, 973e98f67ac548d2bb519dfcad1d7a13; no voters:
I20260924 13:42:26.520198 21963 raft_consensus.cc:2781] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:26.520272 21963 raft_consensus.cc:482] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:26.520303 21963 raft_consensus.cc:3037] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:26.521060 21963 raft_consensus.cc:504] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:26.521200 21963 leader_election.cc:288] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [CANDIDATE]: Term 1 election: Requested vote from peers 11f630c14fc440d689fb5e5f6ca351e5 (127.19.53.193:44363), 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157)
I20260924 13:42:26.521351 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21592b0b643e4358b6777c4bd36ea6ed" candidate_uuid: "973e98f67ac548d2bb519dfcad1d7a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f630c14fc440d689fb5e5f6ca351e5"
I20260924 13:42:26.521379 21694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21592b0b643e4358b6777c4bd36ea6ed" candidate_uuid: "973e98f67ac548d2bb519dfcad1d7a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c"
I20260924 13:42:26.521418 21554 raft_consensus.cc:3037] T 21592b0b643e4358b6777c4bd36ea6ed P 11f630c14fc440d689fb5e5f6ca351e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:26.521435 21694 raft_consensus.cc:3037] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:26.522102 21694 raft_consensus.cc:2454] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 973e98f67ac548d2bb519dfcad1d7a13 in term 1.
I20260924 13:42:26.522179 21554 raft_consensus.cc:2454] T 21592b0b643e4358b6777c4bd36ea6ed P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 973e98f67ac548d2bb519dfcad1d7a13 in term 1.
I20260924 13:42:26.522258 21767 leader_election.cc:302] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27c3dbce756e4e65a07fcd29bbd40f0c, 973e98f67ac548d2bb519dfcad1d7a13; no voters:
I20260924 13:42:26.522356 21963 raft_consensus.cc:2781] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:26.522418 21963 raft_consensus.cc:686] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 LEADER]: Becoming Leader. State: Replica: 973e98f67ac548d2bb519dfcad1d7a13, State: Running, Role: LEADER
I20260924 13:42:26.522486 21963 consensus_queue.cc:237] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } }
I20260924 13:42:26.523069 21396 catalog_manager.cc:5495] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 973e98f67ac548d2bb519dfcad1d7a13 (127.19.53.195). New cstate: current_term: 1 leader_uuid: "973e98f67ac548d2bb519dfcad1d7a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "973e98f67ac548d2bb519dfcad1d7a13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 33073 } health_report { overall_health: HEALTHY } } }
I20260924 13:42:26.530277 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:26.527995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.19.53.195:33073)
0924 13:42:26.528087 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:26.528089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:26.528127 (+ 38us) server_negotiation.cc:407] Connection header received
0924 13:42:26.528194 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:26.528195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:26.528204 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:26.528305 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:26.528422 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:26.528910 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:26.529662 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:26.529847 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:26.529900 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:26.530054 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:26.530081 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:26.530122 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:26.530168 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 13:42:26.530184 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":55,"threads_started":1}
I20260924 13:42:26.534440 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:26.532418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:59349 (local address 127.19.53.195:33073)
0924 13:42:26.532486 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:26.532488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:26.532601 (+ 113us) server_negotiation.cc:407] Connection header received
0924 13:42:26.532639 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:26.532639 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:26.532647 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:26.532692 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:26.532775 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:26.533210 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:26.533892 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:26.534096 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:26.534147 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:26.534252 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:26.534273 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:26.534310 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:26.534346 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 13:42:26.534368 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":51}
I20260924 13:42:26.534456 22013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:26.532364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:33073 (local address 127.19.53.195:59349)
0924 13:42:26.532527 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:26.532553 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:26.532601 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:26.532703 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:26.532704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:26.532705 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:42:26.532738 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:26.532739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:26.533266 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:26.533266 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:26.533829 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:26.533830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:26.534155 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:26.534156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:26.534194 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:26.534197 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:42:26.534314 (+ 117us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:42:26.534315 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:42:26.534345 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 13:42:26.534382 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260924 13:42:26.536514 21694 raft_consensus.cc:1261] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Refusing update from remote peer 973e98f67ac548d2bb519dfcad1d7a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:26.536537 21554 raft_consensus.cc:1261] T 21592b0b643e4358b6777c4bd36ea6ed P 11f630c14fc440d689fb5e5f6ca351e5 [term 1 FOLLOWER]: Refusing update from remote peer 973e98f67ac548d2bb519dfcad1d7a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:26.536700 21939 consensus_queue.cc:1035] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27c3dbce756e4e65a07fcd29bbd40f0c" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:26.536790 21968 consensus_queue.cc:1035] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11f630c14fc440d689fb5e5f6ca351e5" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 44363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 13:42:26.540772 21961 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 13:42:28.058388 21397 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:28.058498 21397 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:28.059267 21397 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9207f6062fbe48b88bc200db53b9794b: Sending DeleteTablet for 3 replicas of tablet 21592b0b643e4358b6777c4bd36ea6ed
I20260924 13:42:28.059508 21813 tablet_service.cc:1514] Processing DeleteTablet for tablet 21592b0b643e4358b6777c4bd36ea6ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:28 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:48272
I20260924 13:42:28.059583 21534 tablet_service.cc:1514] Processing DeleteTablet for tablet 21592b0b643e4358b6777c4bd36ea6ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:28 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:56618
I20260924 13:42:28.059756 21671 tablet_service.cc:1514] Processing DeleteTablet for tablet 21592b0b643e4358b6777c4bd36ea6ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:28 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:52366
I20260924 13:42:28.059719 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 21469
I20260924 13:42:28.060527 22037 tablet_replica.cc:330] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13: stopping tablet replica
I20260924 13:42:28.060628 22037 raft_consensus.cc:2229] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:42:28.060782 22037 raft_consensus.cc:2258] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:28.061092 22037 ts_tablet_manager.cc:1905] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:28.063141 22037 ts_tablet_manager.cc:1918] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 13:42:28.063216 22037 log.cc:1196] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/21592b0b643e4358b6777c4bd36ea6ed
I20260924 13:42:28.063446 22037 ts_tablet_manager.cc:1939] T 21592b0b643e4358b6777c4bd36ea6ed P 973e98f67ac548d2bb519dfcad1d7a13: Deleting consensus metadata
I20260924 13:42:28.063781 21381 catalog_manager.cc:4841] TS 973e98f67ac548d2bb519dfcad1d7a13 (127.19.53.195:33073): tablet 21592b0b643e4358b6777c4bd36ea6ed (table test-table [id=86f10643cdf64e69a39a716f498591f7]) successfully deleted
I20260924 13:42:28.064368 22039 tablet_replica.cc:330] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c: stopping tablet replica
I20260924 13:42:28.064448 22039 raft_consensus.cc:2229] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:28.064503 22039 raft_consensus.cc:2258] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:28.064734 22039 ts_tablet_manager.cc:1905] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:28.065901 22039 ts_tablet_manager.cc:1918] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 13:42:28.065994 22039 log.cc:1196] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/21592b0b643e4358b6777c4bd36ea6ed
I20260924 13:42:28.066220 22039 ts_tablet_manager.cc:1939] T 21592b0b643e4358b6777c4bd36ea6ed P 27c3dbce756e4e65a07fcd29bbd40f0c: Deleting consensus metadata
I20260924 13:42:28.066507 21383 catalog_manager.cc:4841] TS 27c3dbce756e4e65a07fcd29bbd40f0c (127.19.53.194:45157): tablet 21592b0b643e4358b6777c4bd36ea6ed (table test-table [id=86f10643cdf64e69a39a716f498591f7]) successfully deleted
W20260924 13:42:28.069109 21383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:44363 (error 108)
I20260924 13:42:28.069283 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 21609
I20260924 13:42:28.070425 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:28.070184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:44363 (local address 127.0.0.1:52036)
0924 13:42:28.070322 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:28.070379 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:44363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
W20260924 13:42:28.070658 21383 catalog_manager.cc:4584] TS 11f630c14fc440d689fb5e5f6ca351e5 (127.19.53.193:44363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21592b0b643e4358b6777c4bd36ea6ed: Network error: Client connection negotiation failed: client connection to 127.19.53.193:44363: connect: Connection refused (error 111)
I20260924 13:42:28.074972 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 21749
I20260924 13:42:28.080660 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 21365
2026-09-24T13:42:28Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6432 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:28 dist-test-slave-tfm9 krb5kdc[22044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:28 dist-test-slave-tfm9 krb5kdc[22044](info): set up 2 sockets
Sep 24 13:42:28 dist-test-slave-tfm9 krb5kdc[22044](info): commencing operation
krb5kdc: starting...
W20260924 13:42:29.135344 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:29 dist-test-slave-tfm9 krb5kdc[22044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:42:29.248639 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:37251
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:37251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:29.327935 22057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:29.328186 22057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:29.328225 22057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:29.329377 22057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:29.329447 22057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:29.329471 22057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:29.329493 22057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:29.329533 22057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:29.331004 22057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:37251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:37251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:29.331272 22057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:29.331485 22057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:42:29.336673 22057 server_base.cc:1034] running on GCE node
W20260924 13:42:29.336592 22065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:29.336591 22063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:29.336617 22062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:29.337127 22057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:29.337442 22057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:29.338606 22057 hybrid_clock.cc:648] HybridClock initialized: now 1790257349338593 us; error 43 us; skew 500 ppm
Sep 24 13:42:29 dist-test-slave-tfm9 krb5kdc[22044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257349, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:29.341157 22057 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:42:29.341593 22057 webserver.cc:460] Webserver started at http://127.19.53.254:39861/ using document root <none> and password file <none>
I20260924 13:42:29.341823 22057 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:29.341890 22057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:29.342054 22057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:29.343175 22057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "2efe197ff51a4737baa8226b690bcb9a"
format_stamp: "Formatted at 2026-09-24 13:42:29 on dist-test-slave-tfm9"
server_key: "29e8395e8165c0b2e2250bbc316f9014"
server_key_iv: "9559a0e3ee7a9d8527ed164bad599004"
server_key_version: "encryptionkey@0"
I20260924 13:42:29.343544 22057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "2efe197ff51a4737baa8226b690bcb9a"
format_stamp: "Formatted at 2026-09-24 13:42:29 on dist-test-slave-tfm9"
server_key: "29e8395e8165c0b2e2250bbc316f9014"
server_key_iv: "9559a0e3ee7a9d8527ed164bad599004"
server_key_version: "encryptionkey@0"
I20260924 13:42:29.344877 22057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:29.345561 22072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:29.345737 22057 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:29.345842 22057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "2efe197ff51a4737baa8226b690bcb9a"
format_stamp: "Formatted at 2026-09-24 13:42:29 on dist-test-slave-tfm9"
server_key: "29e8395e8165c0b2e2250bbc316f9014"
server_key_iv: "9559a0e3ee7a9d8527ed164bad599004"
server_key_version: "encryptionkey@0"
I20260924 13:42:29.345935 22057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:29.374994 22057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:29.377301 22057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:29.377462 22057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:29.381304 22057 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:37251
I20260924 13:42:29.381351 22124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:37251 every 8 connection(s)
I20260924 13:42:29.381703 22057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:29.382536 22125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:29.382929 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 22057
I20260924 13:42:29.383018 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:29.383216 19671 external_mini_cluster.cc:1428] Setting key 03c21374ab4fea98c80f21961b45ba3e
I20260924 13:42:29.385862 22125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a: Bootstrap starting.
I20260924 13:42:29.386617 22125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:29.386919 22125 log.cc:824] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:29.387611 22125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a: No bootstrap required, opened a new log
Sep 24 13:42:29 dist-test-slave-tfm9 krb5kdc[22044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:29.390190 22125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efe197ff51a4737baa8226b690bcb9a" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37251 } }
I20260924 13:42:29.390352 22125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:29.390493 22125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efe197ff51a4737baa8226b690bcb9a, State: Initialized, Role: FOLLOWER
I20260924 13:42:29.390657 22125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efe197ff51a4737baa8226b690bcb9a" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37251 } }
I20260924 13:42:29.390785 22125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:29.390832 22125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:29.390934 22125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:29.391606 22125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efe197ff51a4737baa8226b690bcb9a" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37251 } }
I20260924 13:42:29.391703 22125 leader_election.cc:302] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2efe197ff51a4737baa8226b690bcb9a; no voters:
I20260924 13:42:29.391878 22125 leader_election.cc:288] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:29.391921 22130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:29.392062 22130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [term 1 LEADER]: Becoming Leader. State: Replica: 2efe197ff51a4737baa8226b690bcb9a, State: Running, Role: LEADER
I20260924 13:42:29.392135 22125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:29.392252 22130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efe197ff51a4737baa8226b690bcb9a" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37251 } }
I20260924 13:42:29.392889 22130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2efe197ff51a4737baa8226b690bcb9a. Latest consensus state: current_term: 1 leader_uuid: "2efe197ff51a4737baa8226b690bcb9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efe197ff51a4737baa8226b690bcb9a" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37251 } } }
I20260924 13:42:29.392894 22132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2efe197ff51a4737baa8226b690bcb9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efe197ff51a4737baa8226b690bcb9a" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37251 } } }
I20260924 13:42:29.393098 22132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:29.393083 22130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:29.393375 22139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:29.393896 22139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:29.394459 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:29.383694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.19.53.254:37251)
0924 13:42:29.383859 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:29.383867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:29.383890 (+ 23us) server_negotiation.cc:407] Connection header received
0924 13:42:29.384307 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:29.384314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:29.384365 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:29.384474 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:29.385209 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.386132 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.387247 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.387739 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.387818 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:29.390329 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:29.390332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:29.390333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:29.390341 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:29.392293 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:29.392690 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:29.392693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:29.392693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:29.392788 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:29.393004 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:29.393006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:29.393006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:29.393326 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:29.393476 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:29.394046 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 13:42:29.394279 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260924 13:42:29.395555 22139 catalog_manager.cc:1347] Generated new cluster ID: 1651195c6e7449859f5cd37631793683
I20260924 13:42:29.395606 22139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:29.407176 22139 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:29.407740 22139 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:29.416323 22139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2efe197ff51a4737baa8226b690bcb9a: Generated new TSK 0
I20260924 13:42:29.416507 22139 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:42:29.441115 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:37251
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:29.520629 22151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:29.520812 22151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:29.520834 22151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:29.521968 22151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:29.522006 22151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:29.522032 22151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:29.523455 22151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:37251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:29.523625 22151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:29.523859 22151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:29.528903 22157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:29.528910 22159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:29.528910 22156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:29.528926 22151 server_base.cc:1034] running on GCE node
I20260924 13:42:29.529371 22151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:29.529625 22151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:29.530769 22151 hybrid_clock.cc:648] HybridClock initialized: now 1790257349530753 us; error 34 us; skew 500 ppm
Sep 24 13:42:29 dist-test-slave-tfm9 krb5kdc[22044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257349, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:29.533288 22151 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:42:29.533754 22151 webserver.cc:460] Webserver started at http://127.19.53.193:34199/ using document root <none> and password file <none>
I20260924 13:42:29.534075 22151 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:29.534140 22151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:29.534271 22151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:29.535372 22151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "e68b49ca375749a0aff93d47e29a6954"
format_stamp: "Formatted at 2026-09-24 13:42:29 on dist-test-slave-tfm9"
server_key: "0680a883a711bc39d60959e45dd70f35"
server_key_iv: "2f9d42305fd764d2f6879a696dc94e65"
server_key_version: "encryptionkey@0"
I20260924 13:42:29.535732 22151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "e68b49ca375749a0aff93d47e29a6954"
format_stamp: "Formatted at 2026-09-24 13:42:29 on dist-test-slave-tfm9"
server_key: "0680a883a711bc39d60959e45dd70f35"
server_key_iv: "2f9d42305fd764d2f6879a696dc94e65"
server_key_version: "encryptionkey@0"
I20260924 13:42:29.537007 22151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:29.537693 22166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:29.537868 22151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:29.537983 22151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "e68b49ca375749a0aff93d47e29a6954"
format_stamp: "Formatted at 2026-09-24 13:42:29 on dist-test-slave-tfm9"
server_key: "0680a883a711bc39d60959e45dd70f35"
server_key_iv: "2f9d42305fd764d2f6879a696dc94e65"
server_key_version: "encryptionkey@0"
I20260924 13:42:29.538069 22151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:29.578598 22151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:29.581043 22151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:29.581213 22151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:29.581480 22151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:29.581809 22151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:29.581863 22151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:29.581903 22151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:29.581943 22151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:29.587554 22151 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:38841
I20260924 13:42:29.587607 22279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:38841 every 8 connection(s)
I20260924 13:42:29.587910 22151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:42:29 dist-test-slave-tfm9 krb5kdc[22044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257349, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:29.595969 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 22151
I20260924 13:42:29.596076 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:42:29.596279 19671 external_mini_cluster.cc:1428] Setting key 2caa82a98d3b9613fc2373ce77fd251f
I20260924 13:42:29.597244 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:29.588587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:33657 (local address 127.19.53.254:37251)
0924 13:42:29.588649 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:29.588651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:29.588853 (+ 202us) server_negotiation.cc:407] Connection header received
0924 13:42:29.589347 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:29.589348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:29.589358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:29.589404 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:29.590359 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.590875 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.591884 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.592117 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.592178 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:29.594377 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:29.594378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:29.594378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:29.594385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:29.595684 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:29.596102 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:29.596103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:29.596104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:29.596168 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:29.596449 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:29.596450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:29.596450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:29.596614 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:29.596688 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:29.597047 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 13:42:29.597131 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:42:29.597380 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:29.588574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:37251 (local address 127.19.53.193:33657)
0924 13:42:29.588732 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:29.588768 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:29.589264 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:29.589466 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:29.589469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:29.590065 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:29.590286 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:29.590291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.590979 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.590980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:29.591809 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:29.591812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.592193 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.592194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:29.592295 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:29.592798 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:29.592805 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:29.594296 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:29.595758 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:29.595764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:29.595766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:29.595983 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:29.596250 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:29.596250 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:29.596250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:29.596388 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:29.596708 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:29.596720 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:29.596873 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 13:42:29.597013 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 13:42:29.597180 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260924 13:42:29.597859 22280 heartbeater.cc:347] Connected to a master server at 127.19.53.254:37251
I20260924 13:42:29.597985 22280 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:29.598165 22280 heartbeater.cc:511] Master 127.19.53.254:37251 requested a full tablet report, sending...
I20260924 13:42:29.598767 22089 ts_manager.cc:194] Registered new tserver with Master: e68b49ca375749a0aff93d47e29a6954 (127.19.53.193:38841)
I20260924 13:42:29.599314 22089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:33657
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:42:29.621775 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:37251
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:29.701546 22285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:29.701795 22285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:29.701828 22285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:29.702962 22285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:29.703049 22285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:29.703092 22285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:29.704552 22285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:37251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:29.704805 22285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:29.705046 22285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:29.709879 22290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:29.709877 22293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:29.709929 22291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:29.710161 22285 server_base.cc:1034] running on GCE node
I20260924 13:42:29.710395 22285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:29.710645 22285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:29.711802 22285 hybrid_clock.cc:648] HybridClock initialized: now 1790257349711795 us; error 52 us; skew 500 ppm
Sep 24 13:42:29 dist-test-slave-tfm9 krb5kdc[22044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257349, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:29.714120 22285 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:42:29.714519 22285 webserver.cc:460] Webserver started at http://127.19.53.194:35051/ using document root <none> and password file <none>
I20260924 13:42:29.714756 22285 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:29.714818 22285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:29.714954 22285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:29.715976 22285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "a00f4867c4db495cb0ebcff5c054eff4"
format_stamp: "Formatted at 2026-09-24 13:42:29 on dist-test-slave-tfm9"
server_key: "fd7396ab2979d4e18ffcb29f7ec7bc4e"
server_key_iv: "3335b29ce0378537ef97682176d4fbff"
server_key_version: "encryptionkey@0"
I20260924 13:42:29.716315 22285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "a00f4867c4db495cb0ebcff5c054eff4"
format_stamp: "Formatted at 2026-09-24 13:42:29 on dist-test-slave-tfm9"
server_key: "fd7396ab2979d4e18ffcb29f7ec7bc4e"
server_key_iv: "3335b29ce0378537ef97682176d4fbff"
server_key_version: "encryptionkey@0"
I20260924 13:42:29.717628 22285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:29.718381 22300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:29.718541 22285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:29.718631 22285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "a00f4867c4db495cb0ebcff5c054eff4"
format_stamp: "Formatted at 2026-09-24 13:42:29 on dist-test-slave-tfm9"
server_key: "fd7396ab2979d4e18ffcb29f7ec7bc4e"
server_key_iv: "3335b29ce0378537ef97682176d4fbff"
server_key_version: "encryptionkey@0"
I20260924 13:42:29.718708 22285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:29.727836 22285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:29.729611 22285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:29.729792 22285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:29.730031 22285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:29.730327 22285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:29.730378 22285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:29.730418 22285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:29.730446 22285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:29.735946 22285 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:41113
I20260924 13:42:29.736016 22413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:41113 every 8 connection(s)
I20260924 13:42:29.736341 22285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:42:29 dist-test-slave-tfm9 krb5kdc[22044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257349, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:29.745016 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:29.737020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:49611 (local address 127.19.53.254:37251)
0924 13:42:29.737064 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:29.737066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:29.737293 (+ 227us) server_negotiation.cc:407] Connection header received
0924 13:42:29.737825 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:29.737827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:29.737838 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:29.737897 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:29.738797 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.739286 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.740255 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.740464 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.740545 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:29.742605 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:29.742607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:29.742607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:29.742615 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:29.743744 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:29.744128 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:29.744129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:29.744130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:29.744186 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:29.744419 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:29.744420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:29.744420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:29.744538 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:29.744612 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:29.744853 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 13:42:29.744927 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:42:29.745189 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:29.736990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:37251 (local address 127.19.53.194:49611)
0924 13:42:29.737180 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:29.737213 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:29.737721 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:29.737986 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:29.737990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:29.738532 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:29.738703 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:29.738707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.739386 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.739387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:29.740169 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:29.740171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.740562 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.740563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:29.740684 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:29.741067 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:29.741075 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:29.742473 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:29.743831 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:29.743837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:29.743839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:29.744043 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:29.744258 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:29.744258 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:29.744259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:29.744354 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:29.744603 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:29.744609 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:29.744733 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 13:42:29.744834 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 13:42:29.744976 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
I20260924 13:42:29.745620 22414 heartbeater.cc:347] Connected to a master server at 127.19.53.254:37251
I20260924 13:42:29.745692 22414 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:29.745810 22414 heartbeater.cc:511] Master 127.19.53.254:37251 requested a full tablet report, sending...
I20260924 13:42:29.746069 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 22285
I20260924 13:42:29.746158 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:42:29.746196 22089 ts_manager.cc:194] Registered new tserver with Master: a00f4867c4db495cb0ebcff5c054eff4 (127.19.53.194:41113)
I20260924 13:42:29.746344 19671 external_mini_cluster.cc:1428] Setting key d759bc810353fecba5d698b554ed9664
I20260924 13:42:29.746589 22089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:49611
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:42:29.770588 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:37251
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:29.850191 22419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:29.850376 22419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:29.850397 22419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:29.851498 22419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:29.851531 22419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:29.851554 22419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:29.852808 22419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:37251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:29.852967 22419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:29.853118 22419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:29.857903 22424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:29.857968 22427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:29.858067 22425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:29.858068 22419 server_base.cc:1034] running on GCE node
I20260924 13:42:29.858287 22419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:29.858521 22419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:29.859688 22419 hybrid_clock.cc:648] HybridClock initialized: now 1790257349859676 us; error 39 us; skew 500 ppm
Sep 24 13:42:29 dist-test-slave-tfm9 krb5kdc[22044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257349, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:29.862164 22419 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:42:29.862613 22419 webserver.cc:460] Webserver started at http://127.19.53.195:40751/ using document root <none> and password file <none>
I20260924 13:42:29.862838 22419 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:29.862903 22419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:29.863054 22419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:29.864058 22419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "43d38342d70e4b64b510ab294bc2a337"
format_stamp: "Formatted at 2026-09-24 13:42:29 on dist-test-slave-tfm9"
server_key: "af454aed17c5f3d9eb39d9073d919c46"
server_key_iv: "2f8ea818da3188e88d266cda9b1a1b0c"
server_key_version: "encryptionkey@0"
I20260924 13:42:29.864423 22419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "43d38342d70e4b64b510ab294bc2a337"
format_stamp: "Formatted at 2026-09-24 13:42:29 on dist-test-slave-tfm9"
server_key: "af454aed17c5f3d9eb39d9073d919c46"
server_key_iv: "2f8ea818da3188e88d266cda9b1a1b0c"
server_key_version: "encryptionkey@0"
I20260924 13:42:29.865731 22419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:29.866510 22434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:29.866732 22419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:29.866845 22419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "43d38342d70e4b64b510ab294bc2a337"
format_stamp: "Formatted at 2026-09-24 13:42:29 on dist-test-slave-tfm9"
server_key: "af454aed17c5f3d9eb39d9073d919c46"
server_key_iv: "2f8ea818da3188e88d266cda9b1a1b0c"
server_key_version: "encryptionkey@0"
I20260924 13:42:29.866933 22419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:29.879573 22419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:29.881284 22419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:29.881387 22419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:29.881542 22419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:29.881822 22419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:29.881855 22419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:29.881875 22419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:29.881947 22419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:29.887533 22419 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:39285
I20260924 13:42:29.887567 22547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:39285 every 8 connection(s)
I20260924 13:42:29.887882 22419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:42:29 dist-test-slave-tfm9 krb5kdc[22044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257349, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:29.895069 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 22419
I20260924 13:42:29.895159 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:42:29.895357 19671 external_mini_cluster.cc:1428] Setting key 856f60c73defd9f3c113f32d17bbb66c
I20260924 13:42:29.897122 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:29.888630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:44451 (local address 127.19.53.254:37251)
0924 13:42:29.888676 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:29.888678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:29.888839 (+ 161us) server_negotiation.cc:407] Connection header received
0924 13:42:29.889301 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:29.889302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:29.889311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:29.889369 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:29.890318 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.890829 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.891783 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.891974 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.892067 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:29.894535 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:29.894537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:29.894537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:29.894543 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:29.895969 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:29.896399 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:29.896400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:29.896400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:29.896450 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:29.896617 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:29.896618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:29.896618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:29.896702 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:29.896747 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:29.896989 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 13:42:29.897062 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:42:29.897305 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:29.888544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:37251 (local address 127.19.53.195:44451)
0924 13:42:29.888727 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:29.888765 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:29.889219 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:29.889429 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:29.889432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:29.890034 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:29.890242 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:29.890247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.890913 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.890913 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:29.891685 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:29.891687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:29.892058 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:29.892059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:29.892204 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:29.892715 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:29.892723 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:29.894436 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:29.896062 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:29.896067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:29.896069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:29.896348 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:29.896491 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:29.896492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:29.896492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:29.896573 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:29.896756 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:29.896760 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:29.896881 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 13:42:29.896960 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 13:42:29.897066 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":116,"threads_started":1}
I20260924 13:42:29.897775 22548 heartbeater.cc:347] Connected to a master server at 127.19.53.254:37251
I20260924 13:42:29.897881 22548 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:29.898059 22548 heartbeater.cc:511] Master 127.19.53.254:37251 requested a full tablet report, sending...
I20260924 13:42:29.898427 22088 ts_manager.cc:194] Registered new tserver with Master: 43d38342d70e4b64b510ab294bc2a337 (127.19.53.195:39285)
I20260924 13:42:29.898798 22088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:44451
I20260924 13:42:29.907095 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 13:42:29.920374 22128 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:29.919582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55788 (local address 127.19.53.254:37251)
0924 13:42:29.919625 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:29.919628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:29.919734 (+ 106us) server_negotiation.cc:407] Connection header received
0924 13:42:29.919787 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:29.919788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:29.919798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:29.919848 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:29.920326 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55788: BlockingRecv error: recv got EOF from 127.0.0.1:55788 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:42:29.921085 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 22151
I20260924 13:42:29.926635 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 22285
I20260924 13:42:29.930827 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 22419
I20260924 13:42:29.934866 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 22057
2026-09-24T13:42:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1852 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:29 dist-test-slave-tfm9 krb5kdc[22560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:29 dist-test-slave-tfm9 krb5kdc[22560](info): set up 2 sockets
Sep 24 13:42:29 dist-test-slave-tfm9 krb5kdc[22560](info): commencing operation
krb5kdc: starting...
W20260924 13:42:31.043237 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:31 dist-test-slave-tfm9 krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:42:31.158250 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:46529
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:45763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:46529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:31.237684 22573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:31.237926 22573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:31.237982 22573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:31.239192 22573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:31.239264 22573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:31.239286 22573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:31.239307 22573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:31.239337 22573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:31.240803 22573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:45763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:46529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:46529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:31.241075 22573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:31.241305 22573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:42:31.245951 22573 server_base.cc:1034] running on GCE node
W20260924 13:42:31.245965 22579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:31.245913 22578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:31.245936 22581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:31.246358 22573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:31.246711 22573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:31.247855 22573 hybrid_clock.cc:648] HybridClock initialized: now 1790257351247846 us; error 31 us; skew 500 ppm
Sep 24 13:42:31 dist-test-slave-tfm9 krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257351, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:31.256150 22573 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:42:31.256516 22573 webserver.cc:460] Webserver started at http://127.19.53.254:35783/ using document root <none> and password file <none>
I20260924 13:42:31.256718 22573 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:31.256757 22573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:31.256844 22573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:31.257809 22573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "2376d59afc7e43a5b28196df5ecf4765"
format_stamp: "Formatted at 2026-09-24 13:42:31 on dist-test-slave-tfm9"
server_key: "38601fd6b1ff9638fef6ee01c233adaa"
server_key_iv: "85cfa5fbcf7bf428ca57a866482893ab"
server_key_version: "encryptionkey@0"
I20260924 13:42:31.258195 22573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "2376d59afc7e43a5b28196df5ecf4765"
format_stamp: "Formatted at 2026-09-24 13:42:31 on dist-test-slave-tfm9"
server_key: "38601fd6b1ff9638fef6ee01c233adaa"
server_key_iv: "85cfa5fbcf7bf428ca57a866482893ab"
server_key_version: "encryptionkey@0"
I20260924 13:42:31.259454 22573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:31.260147 22588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:31.260309 22573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:31.260412 22573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "2376d59afc7e43a5b28196df5ecf4765"
format_stamp: "Formatted at 2026-09-24 13:42:31 on dist-test-slave-tfm9"
server_key: "38601fd6b1ff9638fef6ee01c233adaa"
server_key_iv: "85cfa5fbcf7bf428ca57a866482893ab"
server_key_version: "encryptionkey@0"
I20260924 13:42:31.260486 22573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:31.283094 22573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:31.284929 22573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:31.285105 22573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:31.289110 22573 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:46529
I20260924 13:42:31.289148 22640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:46529 every 8 connection(s)
I20260924 13:42:31.289645 22573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:31.290308 22641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:31.292500 22641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765: Bootstrap starting.
I20260924 13:42:31.292655 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 22573
I20260924 13:42:31.292735 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:31.292909 19671 external_mini_cluster.cc:1428] Setting key 124a35fc9bd5bc12d4dcc42be8198780
I20260924 13:42:31.293244 22641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:31.293608 22641 log.cc:824] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:31.294380 22641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765: No bootstrap required, opened a new log
I20260924 13:42:31.296644 22641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2376d59afc7e43a5b28196df5ecf4765" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 46529 } }
I20260924 13:42:31.296808 22641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:31.296836 22641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2376d59afc7e43a5b28196df5ecf4765, State: Initialized, Role: FOLLOWER
I20260924 13:42:31.296948 22641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2376d59afc7e43a5b28196df5ecf4765" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 46529 } }
I20260924 13:42:31.297024 22641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:31.297065 22641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:31.297119 22641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:31.297669 22641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2376d59afc7e43a5b28196df5ecf4765" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 46529 } }
I20260924 13:42:31.297787 22641 leader_election.cc:302] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2376d59afc7e43a5b28196df5ecf4765; no voters:
I20260924 13:42:31.297987 22641 leader_election.cc:288] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:31.298048 22646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:31.298224 22646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [term 1 LEADER]: Becoming Leader. State: Replica: 2376d59afc7e43a5b28196df5ecf4765, State: Running, Role: LEADER
I20260924 13:42:31.298271 22641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:31.298328 22646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2376d59afc7e43a5b28196df5ecf4765" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 46529 } }
I20260924 13:42:31.298727 22648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2376d59afc7e43a5b28196df5ecf4765. Latest consensus state: current_term: 1 leader_uuid: "2376d59afc7e43a5b28196df5ecf4765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2376d59afc7e43a5b28196df5ecf4765" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 46529 } } }
I20260924 13:42:31.298727 22647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2376d59afc7e43a5b28196df5ecf4765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2376d59afc7e43a5b28196df5ecf4765" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 46529 } } }
I20260924 13:42:31.298837 22648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:31.298843 22647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:31.299319 22651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:31.299890 22651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 13:42:31 dist-test-slave-tfm9 krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:31.302932 22651 catalog_manager.cc:1347] Generated new cluster ID: e74834798e244631af00c11f0a63b5d0
I20260924 13:42:31.302984 22651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:31.303817 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:31.293378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.19.53.254:46529)
0924 13:42:31.293563 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:31.293570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:31.293594 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:42:31.294120 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:31.294128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:31.294183 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:31.294299 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:31.295081 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:31.295953 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:31.297079 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:31.297557 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:31.297639 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:31.300596 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:31.300599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:31.300599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:31.300609 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:31.302264 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:31.302679 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:31.302682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:31.302683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:31.302805 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:31.303028 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:31.303029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:31.303029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:31.303261 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:31.303366 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:31.303494 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 13:42:31.303591 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260924 13:42:31.304687 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:42:31.326640 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:31.322821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.19.53.254:46529)
0924 13:42:31.322876 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:31.322878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:31.322887 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:31.322908 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:31.322909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:31.322919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:31.322975 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:31.323374 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:31.324132 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:31.325075 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:31.325409 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:31.325495 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:31.326182 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:31.326183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:31.326184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:31.326390 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:31.326406 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:31.326412 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:31.326473 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:31.326540 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 13:42:31.326564 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 13:42:31.327826 22651 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:31.328307 22651 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:31.336175 22651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2376d59afc7e43a5b28196df5ecf4765: Generated new TSK 0
I20260924 13:42:31.336356 22651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:42:31.340724 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 22573
2026-09-24T13:42:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1404 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:31 dist-test-slave-tfm9 krb5kdc[22674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:31 dist-test-slave-tfm9 krb5kdc[22674](info): set up 2 sockets
Sep 24 13:42:31 dist-test-slave-tfm9 krb5kdc[22674](info): commencing operation
krb5kdc: starting...
W20260924 13:42:32.367398 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:32 dist-test-slave-tfm9 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:42:32.482812 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:44315
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:44315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:32.561216 22687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:32.561458 22687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:32.561494 22687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:32.562885 22687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:32.562968 22687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:32.562995 22687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:32.563028 22687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:32.563054 22687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:32.564678 22687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:44315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:44315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:32.564944 22687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:32.565160 22687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:32.569994 22692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:32.570047 22693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:32.570152 22687 server_base.cc:1034] running on GCE node
W20260924 13:42:32.570060 22695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:32.570552 22687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:32.570871 22687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:32.572127 22687 hybrid_clock.cc:648] HybridClock initialized: now 1790257352572099 us; error 40 us; skew 500 ppm
Sep 24 13:42:32 dist-test-slave-tfm9 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257352, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:32.574615 22687 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:42:32.575112 22687 webserver.cc:460] Webserver started at http://127.19.53.254:37865/ using document root <none> and password file <none>
I20260924 13:42:32.575338 22687 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:32.575404 22687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:32.575556 22687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:32.576645 22687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "9d390389f30d4bdc9750485f5338bab4"
format_stamp: "Formatted at 2026-09-24 13:42:32 on dist-test-slave-tfm9"
server_key: "541fa66687646d0741f1ba90be071b2b"
server_key_iv: "c9d66033e58c9f73f346a205d9b70bd1"
server_key_version: "encryptionkey@0"
I20260924 13:42:32.577008 22687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "9d390389f30d4bdc9750485f5338bab4"
format_stamp: "Formatted at 2026-09-24 13:42:32 on dist-test-slave-tfm9"
server_key: "541fa66687646d0741f1ba90be071b2b"
server_key_iv: "c9d66033e58c9f73f346a205d9b70bd1"
server_key_version: "encryptionkey@0"
I20260924 13:42:32.578378 22687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:32.579138 22702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:32.579305 22687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:32.579411 22687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "9d390389f30d4bdc9750485f5338bab4"
format_stamp: "Formatted at 2026-09-24 13:42:32 on dist-test-slave-tfm9"
server_key: "541fa66687646d0741f1ba90be071b2b"
server_key_iv: "c9d66033e58c9f73f346a205d9b70bd1"
server_key_version: "encryptionkey@0"
I20260924 13:42:32.579483 22687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:32.586553 22687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:32.588234 22687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:32.588389 22687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:32.592361 22687 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:44315
I20260924 13:42:32.592397 22754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:44315 every 8 connection(s)
I20260924 13:42:32.592725 22687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:32.593680 22755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:32.595901 22755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4: Bootstrap starting.
I20260924 13:42:32.596396 22755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:32.596624 22755 log.cc:824] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:32.596966 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 22687
I20260924 13:42:32.597095 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:32.597262 19671 external_mini_cluster.cc:1428] Setting key 7e358c4cad4e472d6bdb90ba942d3101
I20260924 13:42:32.597479 22755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4: No bootstrap required, opened a new log
I20260924 13:42:32.599634 22755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d390389f30d4bdc9750485f5338bab4" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44315 } }
I20260924 13:42:32.599800 22755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:32.599838 22755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d390389f30d4bdc9750485f5338bab4, State: Initialized, Role: FOLLOWER
I20260924 13:42:32.599993 22755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d390389f30d4bdc9750485f5338bab4" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44315 } }
I20260924 13:42:32.600076 22755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:32.600106 22755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:32.600148 22755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:32.600809 22755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d390389f30d4bdc9750485f5338bab4" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44315 } }
I20260924 13:42:32.600922 22755 leader_election.cc:302] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d390389f30d4bdc9750485f5338bab4; no voters:
I20260924 13:42:32.601058 22755 leader_election.cc:288] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:32.601114 22760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:32.601264 22755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:32.601282 22760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [term 1 LEADER]: Becoming Leader. State: Replica: 9d390389f30d4bdc9750485f5338bab4, State: Running, Role: LEADER
I20260924 13:42:32.601377 22760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d390389f30d4bdc9750485f5338bab4" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44315 } }
I20260924 13:42:32.602401 22762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d390389f30d4bdc9750485f5338bab4. Latest consensus state: current_term: 1 leader_uuid: "9d390389f30d4bdc9750485f5338bab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d390389f30d4bdc9750485f5338bab4" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44315 } } }
I20260924 13:42:32.602499 22762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:32.602829 22761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d390389f30d4bdc9750485f5338bab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d390389f30d4bdc9750485f5338bab4" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44315 } } }
I20260924 13:42:32.603024 22761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4 [sys.catalog]: This master's current role is: LEADER
W20260924 13:42:32.603186 22776 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:32.603281 22776 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:32.603346 22775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:32.603507 22775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:32.604852 22775 catalog_manager.cc:1347] Generated new cluster ID: cb086102f0044466b684c96003912386
I20260924 13:42:32.604902 22775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:42:32 dist-test-slave-tfm9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:32.609484 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:32.597763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.19.53.254:44315)
0924 13:42:32.597995 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:32.598004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:32.598039 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:42:32.598841 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:32.598850 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:32.598909 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:32.599076 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:32.599642 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:32.601312 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:32.602484 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:32.603105 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:32.603211 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:32.605931 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:32.605935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:32.605935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:32.605946 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:32.607797 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:32.608225 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:32.608228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:32.608228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:32.608326 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:32.608549 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:32.608550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:32.608550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:32.608826 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:32.608978 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:32.609178 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 13:42:32.609305 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":106,"threads_started":1}
I20260924 13:42:32.610056 22775 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:32.610548 22775 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:32.620234 22775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d390389f30d4bdc9750485f5338bab4: Generated new TSK 0
I20260924 13:42:32.620427 22775 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:42:32.646795 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:44315
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:32.725697 22781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:32.725890 22781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:32.725958 22781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:32.727110 22781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:32.727176 22781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:32.727221 22781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:32.728679 22781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:44315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:32.728935 22781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:32.729154 22781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:32.733804 22786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:32.733814 22789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:32.733814 22787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:32.734046 22781 server_base.cc:1034] running on GCE node
I20260924 13:42:32.734247 22781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:32.734483 22781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:32.735646 22781 hybrid_clock.cc:648] HybridClock initialized: now 1790257352735640 us; error 34 us; skew 500 ppm
Sep 24 13:42:32 dist-test-slave-tfm9 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257352, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:32.738055 22781 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:42:32.738507 22781 webserver.cc:460] Webserver started at http://127.19.53.193:35811/ using document root <none> and password file <none>
I20260924 13:42:32.738749 22781 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:32.738816 22781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:32.738948 22781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:32.739954 22781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "043f139e3aa341878c75b33e165c8e75"
format_stamp: "Formatted at 2026-09-24 13:42:32 on dist-test-slave-tfm9"
server_key: "4394b4068e7d2325429741c1847e2621"
server_key_iv: "01f22139f4bb7a4beb964d95c3830c3b"
server_key_version: "encryptionkey@0"
I20260924 13:42:32.740329 22781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "043f139e3aa341878c75b33e165c8e75"
format_stamp: "Formatted at 2026-09-24 13:42:32 on dist-test-slave-tfm9"
server_key: "4394b4068e7d2325429741c1847e2621"
server_key_iv: "01f22139f4bb7a4beb964d95c3830c3b"
server_key_version: "encryptionkey@0"
I20260924 13:42:32.741617 22781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:32.742486 22796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:32.742671 22781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:42:32.742766 22781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "043f139e3aa341878c75b33e165c8e75"
format_stamp: "Formatted at 2026-09-24 13:42:32 on dist-test-slave-tfm9"
server_key: "4394b4068e7d2325429741c1847e2621"
server_key_iv: "01f22139f4bb7a4beb964d95c3830c3b"
server_key_version: "encryptionkey@0"
I20260924 13:42:32.742849 22781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:32.750840 22781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:32.752578 22781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:32.752738 22781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:32.752952 22781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:32.753247 22781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:32.753299 22781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:32.753340 22781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:32.753363 22781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:32.759261 22781 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:40467
I20260924 13:42:32.759310 22909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:40467 every 8 connection(s)
I20260924 13:42:32.759614 22781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:42:32.761041 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 22781
I20260924 13:42:32.761153 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:42:32.761353 19671 external_mini_cluster.cc:1428] Setting key 69be9e2ca457090f68bd6bebae540c0b
Sep 24 13:42:32 dist-test-slave-tfm9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257352, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:32.769824 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:32.760284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:41793 (local address 127.19.53.254:44315)
0924 13:42:32.760367 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:32.760370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:32.760538 (+ 168us) server_negotiation.cc:407] Connection header received
0924 13:42:32.761064 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:32.761065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:32.761078 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:32.761134 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:32.762314 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:32.762866 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:32.763977 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:32.764200 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:32.764289 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:32.766912 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:32.766914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:32.766914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:32.766920 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:32.768338 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:32.768839 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:32.768840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:32.768840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:32.768889 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:32.769169 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:32.769169 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:32.769169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:32.769272 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:32.769323 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:32.769685 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 13:42:32.769761 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:42:32.770123 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:32.760238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:44315 (local address 127.19.53.193:41793)
0924 13:42:32.760419 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:32.760453 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:32.760949 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:32.761206 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:32.761209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:32.761905 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:32.762210 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:32.762214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:32.762964 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:32.762964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:32.763871 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:32.763874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:32.764353 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:32.764354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:32.764501 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:32.765039 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:32.765047 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:32.766818 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:32.768426 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:32.768433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:32.768435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:32.768763 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:32.768954 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:32.768955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:32.768956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:32.769102 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:32.769339 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:32.769346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:32.769531 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 13:42:32.769670 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 13:42:32.769856 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
I20260924 13:42:32.770726 22910 heartbeater.cc:347] Connected to a master server at 127.19.53.254:44315
I20260924 13:42:32.770828 22910 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:32.771029 22910 heartbeater.cc:511] Master 127.19.53.254:44315 requested a full tablet report, sending...
I20260924 13:42:32.771628 22719 ts_manager.cc:194] Registered new tserver with Master: 043f139e3aa341878c75b33e165c8e75 (127.19.53.193:40467)
I20260924 13:42:32.772343 22719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:41793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:42:32.788683 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:44315
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:32.871183 22915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:32.871387 22915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:32.871409 22915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:32.872545 22915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:32.872586 22915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:32.872643 22915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:32.874193 22915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:44315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:32.874423 22915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:32.874655 22915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:32.879504 22921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:32.879515 22920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:32.879515 22923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:32.879698 22915 server_base.cc:1034] running on GCE node
I20260924 13:42:32.879899 22915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:32.880155 22915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:32.881320 22915 hybrid_clock.cc:648] HybridClock initialized: now 1790257352881275 us; error 58 us; skew 500 ppm
Sep 24 13:42:32 dist-test-slave-tfm9 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257352, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:32.884063 22915 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:42:32.884518 22915 webserver.cc:460] Webserver started at http://127.19.53.194:41973/ using document root <none> and password file <none>
I20260924 13:42:32.884755 22915 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:32.884821 22915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:32.884970 22915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:32.886057 22915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "ec00b23cbb904574b81b4069d13c0f2f"
format_stamp: "Formatted at 2026-09-24 13:42:32 on dist-test-slave-tfm9"
server_key: "84cad7ed6605a6c4905e2ead2b6fd8ec"
server_key_iv: "12af871436a9f162bf1d19393c22e543"
server_key_version: "encryptionkey@0"
I20260924 13:42:32.886427 22915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "ec00b23cbb904574b81b4069d13c0f2f"
format_stamp: "Formatted at 2026-09-24 13:42:32 on dist-test-slave-tfm9"
server_key: "84cad7ed6605a6c4905e2ead2b6fd8ec"
server_key_iv: "12af871436a9f162bf1d19393c22e543"
server_key_version: "encryptionkey@0"
I20260924 13:42:32.887712 22915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:32.888404 22930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:32.888577 22915 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:32.888669 22915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "ec00b23cbb904574b81b4069d13c0f2f"
format_stamp: "Formatted at 2026-09-24 13:42:32 on dist-test-slave-tfm9"
server_key: "84cad7ed6605a6c4905e2ead2b6fd8ec"
server_key_iv: "12af871436a9f162bf1d19393c22e543"
server_key_version: "encryptionkey@0"
I20260924 13:42:32.888751 22915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:32.908252 22915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:32.910084 22915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:32.910252 22915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:32.910485 22915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:32.910794 22915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:32.910846 22915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:32.910887 22915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:32.910915 22915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:32.916698 22915 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:45831
I20260924 13:42:32.916749 23043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:45831 every 8 connection(s)
I20260924 13:42:32.917057 22915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:42:32 dist-test-slave-tfm9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257352, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:32.923539 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 22915
I20260924 13:42:32.923664 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:42:32.923846 19671 external_mini_cluster.cc:1428] Setting key aee0fdc74c2f8ceeba7404870145f2c6
I20260924 13:42:32.926120 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:32.917795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:39931 (local address 127.19.53.254:44315)
0924 13:42:32.917861 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:32.917864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:32.918031 (+ 167us) server_negotiation.cc:407] Connection header received
0924 13:42:32.918503 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:32.918504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:32.918525 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:32.918582 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:32.919458 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:32.919946 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:32.920898 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:32.921071 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:32.921120 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:32.923118 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:32.923119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:32.923120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:32.923126 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:32.924584 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:32.925060 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:32.925061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:32.925061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:32.925110 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:32.925393 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:32.925394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:32.925394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:32.925487 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:32.925546 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:32.925875 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 13:42:32.925996 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 13:42:32.926301 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:32.917743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:44315 (local address 127.19.53.194:39931)
0924 13:42:32.917890 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:32.917954 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:32.918423 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:32.918635 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:32.918639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:32.919190 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:32.919381 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:32.919385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:32.920023 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:32.920024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:32.920821 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:32.920824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:32.921153 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:32.921154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:32.921252 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:32.921680 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:32.921687 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:32.923031 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:32.924658 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:32.924665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:32.924667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:32.924980 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:32.925188 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:32.925189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:32.925189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:32.925331 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:32.925558 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:32.925563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:32.925731 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 13:42:32.925842 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 13:42:32.926043 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260924 13:42:32.926914 23044 heartbeater.cc:347] Connected to a master server at 127.19.53.254:44315
I20260924 13:42:32.927037 23044 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:32.927223 23044 heartbeater.cc:511] Master 127.19.53.254:44315 requested a full tablet report, sending...
I20260924 13:42:32.927558 22719 ts_manager.cc:194] Registered new tserver with Master: ec00b23cbb904574b81b4069d13c0f2f (127.19.53.194:45831)
I20260924 13:42:32.928081 22719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:39931
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:42:32.948675 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:44315
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:33.028173 23049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:33.028379 23049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:33.028403 23049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:33.029564 23049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:33.029604 23049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:33.029631 23049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:33.031126 23049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:44315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:33.031374 23049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:33.031606 23049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:33.036475 23055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:33.036473 23054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:33.036470 23057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:33.036774 23049 server_base.cc:1034] running on GCE node
I20260924 13:42:33.036995 23049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:33.037246 23049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:33.038401 23049 hybrid_clock.cc:648] HybridClock initialized: now 1790257353038380 us; error 43 us; skew 500 ppm
Sep 24 13:42:33 dist-test-slave-tfm9 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257353, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:33.040927 23049 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:42:33.041353 23049 webserver.cc:460] Webserver started at http://127.19.53.195:33601/ using document root <none> and password file <none>
I20260924 13:42:33.041612 23049 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:33.041657 23049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:33.041821 23049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:33.042838 23049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "6e83e2a253eb4bcbb8d34e3c0254c4ea"
format_stamp: "Formatted at 2026-09-24 13:42:33 on dist-test-slave-tfm9"
server_key: "6cb0beec8d5aeab10b95e55db19b8083"
server_key_iv: "f1afaf09cd1f068a9732898dec90caa8"
server_key_version: "encryptionkey@0"
I20260924 13:42:33.043181 23049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "6e83e2a253eb4bcbb8d34e3c0254c4ea"
format_stamp: "Formatted at 2026-09-24 13:42:33 on dist-test-slave-tfm9"
server_key: "6cb0beec8d5aeab10b95e55db19b8083"
server_key_iv: "f1afaf09cd1f068a9732898dec90caa8"
server_key_version: "encryptionkey@0"
I20260924 13:42:33.044574 23049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:33.045322 23064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:33.045532 23049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:33.045653 23049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "6e83e2a253eb4bcbb8d34e3c0254c4ea"
format_stamp: "Formatted at 2026-09-24 13:42:33 on dist-test-slave-tfm9"
server_key: "6cb0beec8d5aeab10b95e55db19b8083"
server_key_iv: "f1afaf09cd1f068a9732898dec90caa8"
server_key_version: "encryptionkey@0"
I20260924 13:42:33.045742 23049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:33.067438 23049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:33.069725 23049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:33.069833 23049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:33.070072 23049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:33.070391 23049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:33.070423 23049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:33.070465 23049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:33.070493 23049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:33.076417 23049 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:39815
I20260924 13:42:33.076463 23177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:39815 every 8 connection(s)
I20260924 13:42:33.076767 23049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:42:33 dist-test-slave-tfm9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257353, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:33.083282 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 23049
I20260924 13:42:33.083405 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:42:33.083616 19671 external_mini_cluster.cc:1428] Setting key 469a94c6a770c09b21bfcf779bb1aaa9
I20260924 13:42:33.085776 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:33.077446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:54879 (local address 127.19.53.254:44315)
0924 13:42:33.077493 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:33.077496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:33.077708 (+ 212us) server_negotiation.cc:407] Connection header received
0924 13:42:33.078253 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:33.078254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:33.078263 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:33.078326 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:33.079245 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:33.079747 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:33.080672 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:33.080846 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:33.080895 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:33.083003 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:33.083004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:33.083004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:33.083010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:33.084756 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:33.085100 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:33.085101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:33.085101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:33.085149 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:33.085310 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:33.085311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:33.085311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:33.085406 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:33.085452 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:33.085657 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 13:42:33.085715 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:42:33.085901 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:33.077432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:44315 (local address 127.19.53.195:54879)
0924 13:42:33.077596 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:33.077631 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:33.078165 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:33.078379 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:33.078382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:33.078910 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:33.079153 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:33.079160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:33.079820 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:33.079821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:33.080607 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:33.080609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:33.080912 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:33.080913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:33.081010 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:33.081415 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:33.081422 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:33.082922 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:33.084815 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:33.084821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:33.084823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:33.085041 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:33.085188 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:33.085188 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:33.085188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:33.085267 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:33.085442 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:33.085444 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:33.085547 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 13:42:33.085617 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:42:33.085721 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260924 13:42:33.086396 23178 heartbeater.cc:347] Connected to a master server at 127.19.53.254:44315
I20260924 13:42:33.086490 23178 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:33.086628 23178 heartbeater.cc:511] Master 127.19.53.254:44315 requested a full tablet report, sending...
I20260924 13:42:33.086941 22719 ts_manager.cc:194] Registered new tserver with Master: 6e83e2a253eb4bcbb8d34e3c0254c4ea (127.19.53.195:39815)
I20260924 13:42:33.087270 22719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:54879
I20260924 13:42:33.095485 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 13:42:33 dist-test-slave-tfm9 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257353, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 13:42:33 dist-test-slave-tfm9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257353, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:33.117878 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:33.110775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.19.53.254:44315)
0924 13:42:33.110829 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:33.110833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:33.110916 (+ 83us) server_negotiation.cc:407] Connection header received
0924 13:42:33.110943 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:33.110944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:33.110953 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:33.111007 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:33.111539 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:33.112027 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:33.112722 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:33.112897 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:33.112948 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:33.115449 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:33.115451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:33.115451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:33.115458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:33.116813 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:33.117170 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:33.117171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:33.117171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:33.117219 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:33.117397 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:33.117398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:33.117398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:33.117547 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:33.117623 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:33.117752 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 13:42:33.117812 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 13:42:33.118186 22719 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36456
I20260924 13:42:33.118718 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 22781
I20260924 13:42:33.123521 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 22915
I20260924 13:42:33.127746 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 23049
I20260924 13:42:33.132138 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 22687
2026-09-24T13:42:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1793 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:33 dist-test-slave-tfm9 krb5kdc[23190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:33 dist-test-slave-tfm9 krb5kdc[23190](info): set up 2 sockets
Sep 24 13:42:33 dist-test-slave-tfm9 krb5kdc[23190](info): commencing operation
krb5kdc: starting...
W20260924 13:42:34.163298 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:34 dist-test-slave-tfm9 krb5kdc[23190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:42:34.276764 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:34841
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:34841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:34.356266 23203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:34.356464 23203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:34.356493 23203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:34.357625 23203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:34.357664 23203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:34.357678 23203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:34.357692 23203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:34.357703 23203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:34.359220 23203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:34841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:34841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:34.359462 23203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:34.359715 23203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:34.364537 23208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:34.364494 23211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:34.364702 23209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:34.364759 23203 server_base.cc:1034] running on GCE node
I20260924 13:42:34.364979 23203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:34.365252 23203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:34.366410 23203 hybrid_clock.cc:648] HybridClock initialized: now 1790257354366392 us; error 27 us; skew 500 ppm
Sep 24 13:42:34 dist-test-slave-tfm9 krb5kdc[23190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257354, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:34.369045 23203 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:42:34.369549 23203 webserver.cc:460] Webserver started at http://127.19.53.254:38713/ using document root <none> and password file <none>
I20260924 13:42:34.369801 23203 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:34.369867 23203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:34.370054 23203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:34.371098 23203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "135a2a12cc6d437a9c1865bfb764717b"
format_stamp: "Formatted at 2026-09-24 13:42:34 on dist-test-slave-tfm9"
server_key: "2654711b71e8ab903dfb478601f58448"
server_key_iv: "84926da3fa1a2aaa11f3f3c36606ea5d"
server_key_version: "encryptionkey@0"
I20260924 13:42:34.371387 23203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "135a2a12cc6d437a9c1865bfb764717b"
format_stamp: "Formatted at 2026-09-24 13:42:34 on dist-test-slave-tfm9"
server_key: "2654711b71e8ab903dfb478601f58448"
server_key_iv: "84926da3fa1a2aaa11f3f3c36606ea5d"
server_key_version: "encryptionkey@0"
I20260924 13:42:34.372614 23203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:42:34.373278 23218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:34.373457 23203 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:34.373536 23203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "135a2a12cc6d437a9c1865bfb764717b"
format_stamp: "Formatted at 2026-09-24 13:42:34 on dist-test-slave-tfm9"
server_key: "2654711b71e8ab903dfb478601f58448"
server_key_iv: "84926da3fa1a2aaa11f3f3c36606ea5d"
server_key_version: "encryptionkey@0"
I20260924 13:42:34.373588 23203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:34.386497 23203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:34.388664 23203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:34.388763 23203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:34.392618 23203 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:34841
I20260924 13:42:34.392649 23270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:34841 every 8 connection(s)
I20260924 13:42:34.393121 23203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:34.393784 23271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:34.395942 23271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b: Bootstrap starting.
I20260924 13:42:34.396490 23271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:34.396772 23271 log.cc:824] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:34.397423 23271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b: No bootstrap required, opened a new log
I20260924 13:42:34.399019 23271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a2a12cc6d437a9c1865bfb764717b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34841 } }
I20260924 13:42:34.399148 23271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:34.399210 23271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135a2a12cc6d437a9c1865bfb764717b, State: Initialized, Role: FOLLOWER
I20260924 13:42:34.399354 23271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a2a12cc6d437a9c1865bfb764717b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34841 } }
I20260924 13:42:34.399413 23271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:34.399451 23271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:34.399500 23271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:34.400408 23271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a2a12cc6d437a9c1865bfb764717b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34841 } }
I20260924 13:42:34.400516 23271 leader_election.cc:302] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 135a2a12cc6d437a9c1865bfb764717b; no voters:
I20260924 13:42:34.400694 23271 leader_election.cc:288] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:34.400758 23274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:34.401252 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 23203
I20260924 13:42:34.401391 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:34.401574 19671 external_mini_cluster.cc:1428] Setting key 0c7e5b315bc281ba17d16dac2bdfae62
I20260924 13:42:34.400935 23271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:34.401834 23274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [term 1 LEADER]: Becoming Leader. State: Replica: 135a2a12cc6d437a9c1865bfb764717b, State: Running, Role: LEADER
I20260924 13:42:34.401953 23274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a2a12cc6d437a9c1865bfb764717b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34841 } }
I20260924 13:42:34.402546 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "135a2a12cc6d437a9c1865bfb764717b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a2a12cc6d437a9c1865bfb764717b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34841 } } }
I20260924 13:42:34.402629 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:34.402891 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135a2a12cc6d437a9c1865bfb764717b. Latest consensus state: current_term: 1 leader_uuid: "135a2a12cc6d437a9c1865bfb764717b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a2a12cc6d437a9c1865bfb764717b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34841 } } }
I20260924 13:42:34.403002 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:34.402908 23285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:34.403540 23285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 13:42:34 dist-test-slave-tfm9 krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:34.409415 23285 catalog_manager.cc:1347] Generated new cluster ID: 754d563219094d928ef12be4599919a4
I20260924 13:42:34.409471 23285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:34.413021 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:34.402236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.19.53.254:34841)
0924 13:42:34.402987 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:34.402997 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:34.403033 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:42:34.403151 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:34.403158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:34.403231 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:34.403382 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:34.404067 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.405441 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.406305 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.406963 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.407087 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:34.409442 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:34.409446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:34.409446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:34.409457 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:34.411422 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:34.411827 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:34.411830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:34.411830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:34.411927 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:34.412140 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:34.412141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:34.412141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:34.412425 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:34.412568 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:34.412696 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 13:42:34.412816 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":72,"threads_started":1}
I20260924 13:42:34.423966 23285 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:34.424443 23285 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:34.435557 23285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135a2a12cc6d437a9c1865bfb764717b: Generated new TSK 0
I20260924 13:42:34.435765 23285 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:42:34.460522 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34841
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:34.541144 23297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:34.541365 23297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:34.541404 23297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:34.542632 23297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:34.542694 23297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:34.542737 23297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:34.544230 23297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:34.544489 23297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:34.544703 23297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:34.549612 23302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:34.549603 23305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:34.549633 23303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:34.549990 23297 server_base.cc:1034] running on GCE node
I20260924 13:42:34.550247 23297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:34.550494 23297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:34.551659 23297 hybrid_clock.cc:648] HybridClock initialized: now 1790257354551639 us; error 37 us; skew 500 ppm
Sep 24 13:42:34 dist-test-slave-tfm9 krb5kdc[23190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257354, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:34.554179 23297 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:42:34.554659 23297 webserver.cc:460] Webserver started at http://127.19.53.193:42143/ using document root <none> and password file <none>
I20260924 13:42:34.554836 23297 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:34.554879 23297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:34.555024 23297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:34.556046 23297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "154ee713d7ea4567a36392053b820ef5"
format_stamp: "Formatted at 2026-09-24 13:42:34 on dist-test-slave-tfm9"
server_key: "abb233a97c51071ea2fef4f4f360c384"
server_key_iv: "a9c3b2d6d1fc01f8bb92851d42f259af"
server_key_version: "encryptionkey@0"
I20260924 13:42:34.556396 23297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "154ee713d7ea4567a36392053b820ef5"
format_stamp: "Formatted at 2026-09-24 13:42:34 on dist-test-slave-tfm9"
server_key: "abb233a97c51071ea2fef4f4f360c384"
server_key_iv: "a9c3b2d6d1fc01f8bb92851d42f259af"
server_key_version: "encryptionkey@0"
I20260924 13:42:34.557706 23297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:42:34.558554 23312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:34.558758 23297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:42:34.558856 23297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "154ee713d7ea4567a36392053b820ef5"
format_stamp: "Formatted at 2026-09-24 13:42:34 on dist-test-slave-tfm9"
server_key: "abb233a97c51071ea2fef4f4f360c384"
server_key_iv: "a9c3b2d6d1fc01f8bb92851d42f259af"
server_key_version: "encryptionkey@0"
I20260924 13:42:34.558939 23297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:34.569232 23297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:34.571309 23297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:34.571471 23297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:34.571699 23297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:34.571995 23297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:34.572049 23297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:34.572090 23297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:34.572118 23297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:34.578123 23297 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:45835
I20260924 13:42:34.578171 23425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:45835 every 8 connection(s)
I20260924 13:42:34.578478 23297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:42:34 dist-test-slave-tfm9 krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257354, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:34.585402 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 23297
I20260924 13:42:34.585486 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:42:34.585690 19671 external_mini_cluster.cc:1428] Setting key 81981983567b2d3488d4deded94ae9ae
I20260924 13:42:34.587944 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:34.579127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:56753 (local address 127.19.53.254:34841)
0924 13:42:34.579194 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:34.579196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:34.579369 (+ 173us) server_negotiation.cc:407] Connection header received
0924 13:42:34.580290 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:34.580291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:34.580301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:34.580354 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:34.580940 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.581438 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.582478 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.582688 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.582744 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:34.584726 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:34.584727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:34.584728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:34.584734 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:34.586487 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:34.586956 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:34.586957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:34.586957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:34.587006 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:34.587277 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:34.587277 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:34.587278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:34.587371 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:34.587420 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:34.587739 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 13:42:34.587799 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 13:42:34.588256 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:34.579123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34841 (local address 127.19.53.193:56753)
0924 13:42:34.579269 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:34.579304 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:34.580195 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:34.580406 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:34.580409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:34.580659 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:34.580845 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:34.580849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.581565 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.581566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:34.582409 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:34.582411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.582761 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.582761 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:34.582857 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:34.583289 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:34.583297 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:34.584612 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:34.586558 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:34.586565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:34.586567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:34.586877 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:34.587066 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:34.587067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:34.587068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:34.587216 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:34.587432 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:34.587437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:34.587605 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 13:42:34.587717 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 13:42:34.587894 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260924 13:42:34.588722 23426 heartbeater.cc:347] Connected to a master server at 127.19.53.254:34841
I20260924 13:42:34.588837 23426 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:34.589026 23426 heartbeater.cc:511] Master 127.19.53.254:34841 requested a full tablet report, sending...
I20260924 13:42:34.589551 23235 ts_manager.cc:194] Registered new tserver with Master: 154ee713d7ea4567a36392053b820ef5 (127.19.53.193:45835)
I20260924 13:42:34.590337 23235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:56753
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:42:34.612450 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34841
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:34.692957 23431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:34.693176 23431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:34.693214 23431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:34.694377 23431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:34.694448 23431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:34.694494 23431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:34.695967 23431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:34.696230 23431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:34.696503 23431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:34.701344 23439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:34.701359 23436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:34.701331 23437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:34.701606 23431 server_base.cc:1034] running on GCE node
I20260924 13:42:34.701823 23431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:34.702087 23431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:34.703267 23431 hybrid_clock.cc:648] HybridClock initialized: now 1790257354703284 us; error 57 us; skew 500 ppm
Sep 24 13:42:34 dist-test-slave-tfm9 krb5kdc[23190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257354, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:34.705713 23431 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:42:34.706148 23431 webserver.cc:460] Webserver started at http://127.19.53.194:44425/ using document root <none> and password file <none>
I20260924 13:42:34.706339 23431 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:34.706378 23431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:34.706471 23431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:34.707495 23431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "3cad51b8b6634456aae5f01e59204e2a"
format_stamp: "Formatted at 2026-09-24 13:42:34 on dist-test-slave-tfm9"
server_key: "566c2d0f8a7e3f1831597db043df0e1b"
server_key_iv: "a6b0364f240b0a0c0039487eed479394"
server_key_version: "encryptionkey@0"
I20260924 13:42:34.707834 23431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "3cad51b8b6634456aae5f01e59204e2a"
format_stamp: "Formatted at 2026-09-24 13:42:34 on dist-test-slave-tfm9"
server_key: "566c2d0f8a7e3f1831597db043df0e1b"
server_key_iv: "a6b0364f240b0a0c0039487eed479394"
server_key_version: "encryptionkey@0"
I20260924 13:42:34.709165 23431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:34.709899 23446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:34.710101 23431 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:34.710186 23431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "3cad51b8b6634456aae5f01e59204e2a"
format_stamp: "Formatted at 2026-09-24 13:42:34 on dist-test-slave-tfm9"
server_key: "566c2d0f8a7e3f1831597db043df0e1b"
server_key_iv: "a6b0364f240b0a0c0039487eed479394"
server_key_version: "encryptionkey@0"
I20260924 13:42:34.710240 23431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:34.732409 23431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:34.735123 23431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:34.735306 23431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:34.735584 23431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:34.735968 23431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:34.736030 23431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:34.736064 23431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:34.736092 23431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:34.742043 23431 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:41451
I20260924 13:42:34.742128 23559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:41451 every 8 connection(s)
I20260924 13:42:34.742430 23431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:42:34.747205 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 23431
I20260924 13:42:34.747310 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:42:34.747486 19671 external_mini_cluster.cc:1428] Setting key 7c460725a05415321b73579a69f52431
Sep 24 13:42:34 dist-test-slave-tfm9 krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257354, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:34.752409 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:34.743197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:33965 (local address 127.19.53.254:34841)
0924 13:42:34.743260 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:34.743262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:34.743466 (+ 204us) server_negotiation.cc:407] Connection header received
0924 13:42:34.744415 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:34.744416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:34.744428 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:34.744484 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:34.745088 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.745592 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.746758 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.746963 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.747046 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:34.749365 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:34.749367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:34.749367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:34.749375 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:34.750874 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:34.751399 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:34.751400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:34.751401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:34.751470 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:34.751754 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:34.751754 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:34.751754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:34.751847 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:34.751893 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:34.752225 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 13:42:34.752345 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:42:34.752692 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:34.743167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34841 (local address 127.19.53.194:33965)
0924 13:42:34.743352 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:34.743387 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:34.744309 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:34.744559 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:34.744563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:34.744807 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:34.744995 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:34.744999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.745681 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.745682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:34.746657 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:34.746659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.747065 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.747066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:34.747210 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:34.747715 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:34.747722 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:34.749273 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:34.750956 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:34.750963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:34.750965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:34.751293 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:34.751537 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:34.751538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:34.751538 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:34.751686 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:34.751910 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:34.751916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:34.752088 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 13:42:34.752210 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 13:42:34.752390 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20260924 13:42:34.753145 23560 heartbeater.cc:347] Connected to a master server at 127.19.53.254:34841
I20260924 13:42:34.753239 23560 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:34.753427 23560 heartbeater.cc:511] Master 127.19.53.254:34841 requested a full tablet report, sending...
I20260924 13:42:34.753799 23235 ts_manager.cc:194] Registered new tserver with Master: 3cad51b8b6634456aae5f01e59204e2a (127.19.53.194:41451)
I20260924 13:42:34.754182 23235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:33965
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:42:34.771555 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34841
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:34.852468 23565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:34.852711 23565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:34.852739 23565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:34.853909 23565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:34.853972 23565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:34.853998 23565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:34.855486 23565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:34.855729 23565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:34.855952 23565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:34.860968 23570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:34.861109 23565 server_base.cc:1034] running on GCE node
W20260924 13:42:34.860973 23571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:34.860975 23573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:34.861462 23565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:34.861750 23565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:34.862903 23565 hybrid_clock.cc:648] HybridClock initialized: now 1790257354862893 us; error 28 us; skew 500 ppm
Sep 24 13:42:34 dist-test-slave-tfm9 krb5kdc[23190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257354, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:34.865268 23565 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:42:34.865700 23565 webserver.cc:460] Webserver started at http://127.19.53.195:39121/ using document root <none> and password file <none>
I20260924 13:42:34.865988 23565 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:34.866060 23565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:34.866199 23565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:34.867226 23565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "49b5f67d542647d1aa9eadc842c85c28"
format_stamp: "Formatted at 2026-09-24 13:42:34 on dist-test-slave-tfm9"
server_key: "8db3a3c67b45ac91a30d3bf11d2dc1bf"
server_key_iv: "88d2bc3fe5c02055cb9b8411d7c31155"
server_key_version: "encryptionkey@0"
I20260924 13:42:34.867591 23565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "49b5f67d542647d1aa9eadc842c85c28"
format_stamp: "Formatted at 2026-09-24 13:42:34 on dist-test-slave-tfm9"
server_key: "8db3a3c67b45ac91a30d3bf11d2dc1bf"
server_key_iv: "88d2bc3fe5c02055cb9b8411d7c31155"
server_key_version: "encryptionkey@0"
I20260924 13:42:34.869052 23565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:34.869778 23580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:34.870000 23565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:34.870102 23565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "49b5f67d542647d1aa9eadc842c85c28"
format_stamp: "Formatted at 2026-09-24 13:42:34 on dist-test-slave-tfm9"
server_key: "8db3a3c67b45ac91a30d3bf11d2dc1bf"
server_key_iv: "88d2bc3fe5c02055cb9b8411d7c31155"
server_key_version: "encryptionkey@0"
I20260924 13:42:34.870186 23565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:34.890282 23565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:34.892114 23565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:34.892280 23565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:34.892529 23565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:34.892858 23565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:34.892913 23565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:34.892954 23565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:34.892984 23565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:34.898821 23565 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:38053
I20260924 13:42:34.898867 23693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:38053 every 8 connection(s)
I20260924 13:42:34.899165 23565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:42:34 dist-test-slave-tfm9 krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257354, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:34.906207 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 23565
I20260924 13:42:34.906319 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:42:34.906531 19671 external_mini_cluster.cc:1428] Setting key a79989ec516f86bb892711db3707eb95
I20260924 13:42:34.908807 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:34.899895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:60625 (local address 127.19.53.254:34841)
0924 13:42:34.899943 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:34.899945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:34.900154 (+ 209us) server_negotiation.cc:407] Connection header received
0924 13:42:34.901071 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:34.901072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:34.901082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:34.901147 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:34.901836 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.902506 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.903683 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.903868 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.903926 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:34.905903 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:34.905904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:34.905904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:34.905947 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:34.907579 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:34.907959 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:34.907960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:34.907960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:34.908028 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:34.908264 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:34.908265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:34.908265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:34.908378 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:34.908437 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:34.908686 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 13:42:34.908743 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:42:34.908977 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:34.899872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34841 (local address 127.19.53.195:60625)
0924 13:42:34.900026 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:34.900066 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:34.900968 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:34.901200 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:34.901203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:34.901449 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:34.901727 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:34.901732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.902590 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.902591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:34.903607 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:34.903609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.903941 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.903942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:34.904040 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:34.904472 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:34.904480 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:34.905815 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:34.907648 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:34.907654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:34.907655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:34.907868 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:34.908095 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:34.908096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:34.908096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:34.908193 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:34.908430 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:34.908432 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:34.908572 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 13:42:34.908656 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 13:42:34.908774 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260924 13:42:34.909416 23694 heartbeater.cc:347] Connected to a master server at 127.19.53.254:34841
I20260924 13:42:34.909528 23694 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:34.909703 23694 heartbeater.cc:511] Master 127.19.53.254:34841 requested a full tablet report, sending...
I20260924 13:42:34.910130 23235 ts_manager.cc:194] Registered new tserver with Master: 49b5f67d542647d1aa9eadc842c85c28 (127.19.53.195:38053)
I20260924 13:42:34.910512 23235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:60625
I20260924 13:42:34.918260 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 13:42:34 dist-test-slave-tfm9 krb5kdc[23190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 13:42:34 dist-test-slave-tfm9 krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.193@KRBTEST.COM
I20260924 13:42:34.942098 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:34.933872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.19.53.193:45835)
0924 13:42:34.934120 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:34.934125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:34.934138 (+ 13us) server_negotiation.cc:407] Connection header received
0924 13:42:34.934165 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:34.934167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:34.934181 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:34.934255 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:34.934921 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.935986 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.936743 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.937089 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.937158 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:34.939482 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:34.939494 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:34.939494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:34.939501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:34.940906 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:34.941268 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:34.941271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:34.941271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:34.941331 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:34.941531 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:34.941531 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:34.941531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:34.941659 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:34.941765 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:34.941892 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 13:42:34.942013 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":125,"threads_started":1}
Sep 24 13:42:34 dist-test-slave-tfm9 krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:34.950045 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:34.943520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.19.53.254:34841)
0924 13:42:34.943588 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:34.943590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:34.943599 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:34.943646 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:34.943647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:34.943656 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:34.943702 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:34.944154 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.944626 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.945277 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:34.945449 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:34.945500 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:34.947602 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:34.947604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:34.947604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:34.947611 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:34.949001 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:34.949310 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:34.949311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:34.949311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:34.949360 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:34.949517 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:34.949518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:34.949518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:34.949650 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:34.949721 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:34.949807 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 13:42:34.949860 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 13:42:34.950317 23235 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38828
I20260924 13:42:34.950688 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 23297
I20260924 13:42:34.955884 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 23431
I20260924 13:42:34.960194 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 23565
I20260924 13:42:34.964504 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 23203
2026-09-24T13:42:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1832 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:35 dist-test-slave-tfm9 krb5kdc[23706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:35 dist-test-slave-tfm9 krb5kdc[23706](info): set up 2 sockets
Sep 24 13:42:35 dist-test-slave-tfm9 krb5kdc[23706](info): commencing operation
krb5kdc: starting...
W20260924 13:42:36.000102 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:36 dist-test-slave-tfm9 krb5kdc[23706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:42:36.118709 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:41079
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:46623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:41079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:36.202152 23719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:36.202358 23719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:36.202382 23719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:36.203670 23719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:36.203711 23719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:36.203725 23719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:36.203738 23719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:36.203750 23719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:36.205243 23719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:46623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:41079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:41079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:36.205485 23719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:36.205725 23719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:36.210438 23725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:36.210464 23724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:36.210464 23727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:36.210649 23719 server_base.cc:1034] running on GCE node
I20260924 13:42:36.210958 23719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:36.211305 23719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:36.212462 23719 hybrid_clock.cc:648] HybridClock initialized: now 1790257356212443 us; error 52 us; skew 500 ppm
Sep 24 13:42:36 dist-test-slave-tfm9 krb5kdc[23706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257356, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:36.214998 23719 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:42:36.215332 23719 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:36.215402 23719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:36.215555 23719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:36.216609 23719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "2893176d2b454a28b076359a806a31c1"
format_stamp: "Formatted at 2026-09-24 13:42:36 on dist-test-slave-tfm9"
server_key: "b92e7c3556518b60f93ce5aa1fa1ad3f"
server_key_iv: "5ab1435c078e0d4b6f3820e53bcfd485"
server_key_version: "encryptionkey@0"
I20260924 13:42:36.216966 23719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "2893176d2b454a28b076359a806a31c1"
format_stamp: "Formatted at 2026-09-24 13:42:36 on dist-test-slave-tfm9"
server_key: "b92e7c3556518b60f93ce5aa1fa1ad3f"
server_key_iv: "5ab1435c078e0d4b6f3820e53bcfd485"
server_key_version: "encryptionkey@0"
I20260924 13:42:36.218292 23719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:36.219043 23733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:36.219223 23719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:36.219329 23719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "2893176d2b454a28b076359a806a31c1"
format_stamp: "Formatted at 2026-09-24 13:42:36 on dist-test-slave-tfm9"
server_key: "b92e7c3556518b60f93ce5aa1fa1ad3f"
server_key_iv: "5ab1435c078e0d4b6f3820e53bcfd485"
server_key_version: "encryptionkey@0"
I20260924 13:42:36.219405 23719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:36.233804 23719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:36.236263 23719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:36.236429 23719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:36.240329 23719 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:41079
I20260924 13:42:36.240366 23785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:41079 every 8 connection(s)
I20260924 13:42:36.240700 23719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:36.241714 23786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:36.243758 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 23719
I20260924 13:42:36.243883 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:36.244091 19671 external_mini_cluster.cc:1428] Setting key 9304561f7c7ba14ad316cf80358b8715
I20260924 13:42:36.244553 23786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1: Bootstrap starting.
I20260924 13:42:36.245199 23786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:36.245560 23786 log.cc:824] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:36.246405 23786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1: No bootstrap required, opened a new log
I20260924 13:42:36.248595 23786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2893176d2b454a28b076359a806a31c1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41079 } }
I20260924 13:42:36.248731 23786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:36.248756 23786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2893176d2b454a28b076359a806a31c1, State: Initialized, Role: FOLLOWER
I20260924 13:42:36.248906 23786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2893176d2b454a28b076359a806a31c1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41079 } }
I20260924 13:42:36.248986 23786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:36.249030 23786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:36.249085 23786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:36.249632 23786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2893176d2b454a28b076359a806a31c1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41079 } }
I20260924 13:42:36.249763 23786 leader_election.cc:302] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2893176d2b454a28b076359a806a31c1; no voters:
I20260924 13:42:36.250005 23786 leader_election.cc:288] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:36.250028 23791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:36.250142 23791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [term 1 LEADER]: Becoming Leader. State: Replica: 2893176d2b454a28b076359a806a31c1, State: Running, Role: LEADER
I20260924 13:42:36.250228 23791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2893176d2b454a28b076359a806a31c1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41079 } }
I20260924 13:42:36.250274 23786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:36.250561 23791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2893176d2b454a28b076359a806a31c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2893176d2b454a28b076359a806a31c1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41079 } } }
I20260924 13:42:36.250639 23791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:36.250777 23791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2893176d2b454a28b076359a806a31c1. Latest consensus state: current_term: 1 leader_uuid: "2893176d2b454a28b076359a806a31c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2893176d2b454a28b076359a806a31c1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41079 } } }
I20260924 13:42:36.250847 23791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:36.250996 23796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 13:42:36 dist-test-slave-tfm9 krb5kdc[23706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:36.252341 23796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:36.254429 23796 catalog_manager.cc:1347] Generated new cluster ID: 79dec1df9f4642719b3aa16ea910564f
I20260924 13:42:36.254489 23796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:36.255927 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.244648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.19.53.254:41079)
0924 13:42:36.244823 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:36.244832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:36.244866 (+ 34us) server_negotiation.cc:407] Connection header received
0924 13:42:36.245358 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:36.245366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:36.245414 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:36.245535 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:36.246430 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.247336 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.248496 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.249045 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.249152 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.252204 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:36.252209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:36.252209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:36.252223 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:36.254338 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:36.254763 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:36.254767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:36.254767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:36.254864 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:36.255084 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:36.255085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:36.255086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:36.255320 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:36.255456 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:36.255578 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 13:42:36.255715 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260924 13:42:36.260672 23796 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:36.261168 23796 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:36.264760 23796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1: Generated new TSK 0
I20260924 13:42:36.264930 23796 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:42:36.290895 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41079
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:46623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:36.372941 23812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:36.373188 23812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:36.373227 23812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:36.374440 23812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:36.374503 23812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:36.374550 23812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:36.376230 23812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:46623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:36.376507 23812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:36.376735 23812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:36.382637 23818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:36.382622 23817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:36.382612 23820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:36.382841 23812 server_base.cc:1034] running on GCE node
I20260924 13:42:36.383129 23812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:36.383401 23812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:36.384569 23812 hybrid_clock.cc:648] HybridClock initialized: now 1790257356384557 us; error 38 us; skew 500 ppm
Sep 24 13:42:36 dist-test-slave-tfm9 krb5kdc[23706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257356, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:36.399111 23812 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:42:36.399456 23812 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:36.399500 23812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:36.399632 23812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:36.400673 23812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "af75683a37c4478d87265bd0ba9476b0"
format_stamp: "Formatted at 2026-09-24 13:42:36 on dist-test-slave-tfm9"
server_key: "7d699cee6fba1932219fc818b670e25c"
server_key_iv: "08551e6074c36e038c624a8018530a69"
server_key_version: "encryptionkey@0"
I20260924 13:42:36.400980 23812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "af75683a37c4478d87265bd0ba9476b0"
format_stamp: "Formatted at 2026-09-24 13:42:36 on dist-test-slave-tfm9"
server_key: "7d699cee6fba1932219fc818b670e25c"
server_key_iv: "08551e6074c36e038c624a8018530a69"
server_key_version: "encryptionkey@0"
I20260924 13:42:36.402334 23812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:36.403093 23826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:36.403295 23812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:36.403390 23812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "af75683a37c4478d87265bd0ba9476b0"
format_stamp: "Formatted at 2026-09-24 13:42:36 on dist-test-slave-tfm9"
server_key: "7d699cee6fba1932219fc818b670e25c"
server_key_iv: "08551e6074c36e038c624a8018530a69"
server_key_version: "encryptionkey@0"
I20260924 13:42:36.403447 23812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:36.425169 23812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:36.428092 23812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:36.428246 23812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:36.428483 23812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:36.428778 23812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:36.428814 23812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:36.428843 23812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:36.428859 23812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:36.435638 23812 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:34969
I20260924 13:42:36.435674 23939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:34969 every 8 connection(s)
I20260924 13:42:36.436012 23812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:42:36.436614 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 23812
I20260924 13:42:36.436726 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:42:36.436952 19671 external_mini_cluster.cc:1428] Setting key 5743b6c4459033180bb5e2329c5ac876
Sep 24 13:42:36 dist-test-slave-tfm9 krb5kdc[23706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257356, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
I20260924 13:42:36.447067 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.436799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:36541 (local address 127.19.53.254:41079)
0924 13:42:36.436881 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:36.436883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:36.437205 (+ 322us) server_negotiation.cc:407] Connection header received
0924 13:42:36.437755 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:36.437756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:36.437772 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:36.437848 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:36.438846 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.439536 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.441123 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.441422 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.441490 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.444077 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:36.444079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:36.444079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:36.444087 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:36.445534 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:36.445999 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:36.446000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:36.446001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:36.446060 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:36.446327 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:36.446328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:36.446328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:36.446450 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:36.446540 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:36.446881 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 13:42:36.446952 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 13:42:36.447193 23942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.436800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:41079 (local address 127.19.53.193:36541)
0924 13:42:36.437036 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:36.437088 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:36.437601 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:36.437902 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:36.437905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:36.438228 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:36.438729 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.438733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.439634 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.439635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.441010 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.441014 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.441527 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.441529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.441683 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.442282 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:36.442293 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:36.443954 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:36.445621 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:36.445629 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:36.445631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:36.445882 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:36.446136 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:36.446137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:36.446138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:36.446253 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:36.446534 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:36.446540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:36.446697 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 13:42:36.446828 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 13:42:36.446975 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":115,"threads_started":1}
I20260924 13:42:36.447774 23940 heartbeater.cc:347] Connected to a master server at 127.19.53.254:41079
I20260924 13:42:36.447885 23940 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:36.448040 23940 heartbeater.cc:511] Master 127.19.53.254:41079 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
I20260924 13:42:36.448673 23750 ts_manager.cc:194] Registered new tserver with Master: af75683a37c4478d87265bd0ba9476b0 (127.19.53.193:34969)
I20260924 13:42:36.449301 23750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:36541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:42:36.464304 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41079
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:46623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:36.543939 23945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:36.544139 23945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:36.544163 23945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:36.545230 23945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:36.545264 23945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:36.545288 23945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:36.546718 23945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:46623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:36.546895 23945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:36.547070 23945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:36.552139 23951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:36.552182 23953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:36.552140 23950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:36.552139 23945 server_base.cc:1034] running on GCE node
I20260924 13:42:36.552556 23945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:36.552800 23945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:36.553978 23945 hybrid_clock.cc:648] HybridClock initialized: now 1790257356553965 us; error 40 us; skew 500 ppm
Sep 24 13:42:36 dist-test-slave-tfm9 krb5kdc[23706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257356, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:36.556440 23945 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:42:36.556798 23945 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:36.556845 23945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:36.557026 23945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:36.558151 23945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "771374348bc04268acc1d684be0fadda"
format_stamp: "Formatted at 2026-09-24 13:42:36 on dist-test-slave-tfm9"
server_key: "9f34742591c1adc8572b581a32a94545"
server_key_iv: "cf87137030aad6ea8bb3109c50e9928a"
server_key_version: "encryptionkey@0"
I20260924 13:42:36.558561 23945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "771374348bc04268acc1d684be0fadda"
format_stamp: "Formatted at 2026-09-24 13:42:36 on dist-test-slave-tfm9"
server_key: "9f34742591c1adc8572b581a32a94545"
server_key_iv: "cf87137030aad6ea8bb3109c50e9928a"
server_key_version: "encryptionkey@0"
I20260924 13:42:36.559864 23945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:36.560644 23959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:36.560830 23945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:36.560918 23945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "771374348bc04268acc1d684be0fadda"
format_stamp: "Formatted at 2026-09-24 13:42:36 on dist-test-slave-tfm9"
server_key: "9f34742591c1adc8572b581a32a94545"
server_key_iv: "cf87137030aad6ea8bb3109c50e9928a"
server_key_version: "encryptionkey@0"
I20260924 13:42:36.560990 23945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:36.572791 23945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:36.574801 23945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:36.574950 23945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:36.575171 23945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:36.575460 23945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:36.575528 23945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:36.575562 23945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:36.575600 23945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:36.581374 23945 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:35323
I20260924 13:42:36.581435 24072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:35323 every 8 connection(s)
I20260924 13:42:36.581692 23945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:42:36 dist-test-slave-tfm9 krb5kdc[23706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257356, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:36.588996 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 23945
I20260924 13:42:36.589115 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:42:36.589375 19671 external_mini_cluster.cc:1428] Setting key b51e5e0fbbeb87e27d01723018836f6f
I20260924 13:42:36.590703 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.582359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:37441 (local address 127.19.53.254:41079)
0924 13:42:36.582400 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:36.582404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:36.582600 (+ 196us) server_negotiation.cc:407] Connection header received
0924 13:42:36.583061 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:36.583062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:36.583070 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:36.583124 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:36.584018 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.584511 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.585494 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.585662 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.585708 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.587694 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:36.587695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:36.587695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:36.587701 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:36.589155 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:36.589674 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:36.589676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:36.589676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:36.589738 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:36.590021 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:36.590022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:36.590023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:36.590141 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:36.590215 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:36.590578 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 13:42:36.590640 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:36.590993 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.582331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:41079 (local address 127.19.53.194:37441)
0924 13:42:36.582479 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:36.582515 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:36.582978 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:36.583213 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:36.583216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:36.583459 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:36.583929 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.583934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.584584 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.584585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.585426 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.585428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.585724 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.585725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.585809 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.586291 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:36.586298 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:36.587611 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:36.589278 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:36.589299 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:36.589301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:36.589592 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:36.589805 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:36.589806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:36.589806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:36.589953 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:36.590248 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:36.590254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:36.590434 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 13:42:36.590558 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 13:42:36.590728 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260924 13:42:36.591451 24073 heartbeater.cc:347] Connected to a master server at 127.19.53.254:41079
I20260924 13:42:36.591557 24073 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:36.591711 24073 heartbeater.cc:511] Master 127.19.53.254:41079 requested a full tablet report, sending...
I20260924 13:42:36.592089 23750 ts_manager.cc:194] Registered new tserver with Master: 771374348bc04268acc1d684be0fadda (127.19.53.194:35323)
I20260924 13:42:36.592450 23750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:37441
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:42:36.612673 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41079
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:46623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:36.691574 24078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:36.691771 24078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:36.691795 24078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:36.692929 24078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:36.692968 24078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:36.693027 24078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:36.694571 24078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:46623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:36.694828 24078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:36.695051 24078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:36.699944 24084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:36.699927 24086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:36.699928 24083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:36.699959 24078 server_base.cc:1034] running on GCE node
I20260924 13:42:36.700438 24078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:36.700680 24078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:36.701821 24078 hybrid_clock.cc:648] HybridClock initialized: now 1790257356701802 us; error 35 us; skew 500 ppm
Sep 24 13:42:36 dist-test-slave-tfm9 krb5kdc[23706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257356, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:36.704193 24078 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:42:36.704520 24078 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:36.704593 24078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:36.704737 24078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:36.706148 24078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "4accd46f8a5f409fbfebe760f926d2f4"
format_stamp: "Formatted at 2026-09-24 13:42:36 on dist-test-slave-tfm9"
server_key: "525135a7ac88e76c34f0e88867318c9d"
server_key_iv: "9e22a37bc489819506272490f3f5b6e7"
server_key_version: "encryptionkey@0"
I20260924 13:42:36.706642 24078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "4accd46f8a5f409fbfebe760f926d2f4"
format_stamp: "Formatted at 2026-09-24 13:42:36 on dist-test-slave-tfm9"
server_key: "525135a7ac88e76c34f0e88867318c9d"
server_key_iv: "9e22a37bc489819506272490f3f5b6e7"
server_key_version: "encryptionkey@0"
I20260924 13:42:36.708209 24078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:42:36.709100 24092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:36.709290 24078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:36.709441 24078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "4accd46f8a5f409fbfebe760f926d2f4"
format_stamp: "Formatted at 2026-09-24 13:42:36 on dist-test-slave-tfm9"
server_key: "525135a7ac88e76c34f0e88867318c9d"
server_key_iv: "9e22a37bc489819506272490f3f5b6e7"
server_key_version: "encryptionkey@0"
I20260924 13:42:36.709532 24078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:36.723392 24078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:36.725283 24078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:36.725468 24078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:36.725705 24078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:36.726159 24078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:36.726230 24078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:36.726279 24078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:36.726313 24078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:36.733350 24078 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:44075
I20260924 13:42:36.733382 24205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:44075 every 8 connection(s)
I20260924 13:42:36.733726 24078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:42:36.736949 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 24078
I20260924 13:42:36.737061 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:42:36.737300 19671 external_mini_cluster.cc:1428] Setting key 787b1f8d86a2cd461edac2a24d1ba6b7
Sep 24 13:42:36 dist-test-slave-tfm9 krb5kdc[23706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257356, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:36.743470 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.734557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:35187 (local address 127.19.53.254:41079)
0924 13:42:36.734640 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:36.734642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:36.734838 (+ 196us) server_negotiation.cc:407] Connection header received
0924 13:42:36.735341 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:36.735342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:36.735351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:36.735407 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:36.736351 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.736928 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.738412 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.738607 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.738662 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.740805 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:36.740806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:36.740806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:36.740812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:36.742299 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:36.742692 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:36.742692 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:36.742693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:36.742741 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:36.742945 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:36.742946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:36.742946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:36.743065 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:36.743114 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:36.743353 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 13:42:36.743409 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 13:42:36.743718 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.734518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:41079 (local address 127.19.53.195:35187)
0924 13:42:36.734691 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:36.734737 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:36.735256 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:36.735465 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:36.735468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:36.735735 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:36.736271 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.736276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.737062 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.737063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.738332 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.738335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.738685 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.738686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.738791 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.739242 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:36.739250 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:36.740715 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:36.742382 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:36.742388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:36.742389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:36.742628 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:36.742798 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:36.742799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:36.742799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:36.742893 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:36.743118 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:36.743122 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:36.743231 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 13:42:36.743323 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 13:42:36.743433 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260924 13:42:36.744314 24206 heartbeater.cc:347] Connected to a master server at 127.19.53.254:41079
I20260924 13:42:36.744395 24206 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:36.744514 24206 heartbeater.cc:511] Master 127.19.53.254:41079 requested a full tablet report, sending...
I20260924 13:42:36.744846 23750 ts_manager.cc:194] Registered new tserver with Master: 4accd46f8a5f409fbfebe760f926d2f4 (127.19.53.195:44075)
I20260924 13:42:36.745219 23750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:35187
I20260924 13:42:36.748946 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:42:36.756203 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.750267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.19.53.254:41079)
0924 13:42:36.750317 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:36.750319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:36.750448 (+ 129us) server_negotiation.cc:407] Connection header received
0924 13:42:36.750516 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:36.750517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:36.750527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:36.750570 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:36.751179 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.751641 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.752472 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.752673 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.752753 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.753824 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:36.753825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:36.753826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:36.753833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:36.754946 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:36.755300 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:36.755302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:36.755302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:36.755357 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:36.755579 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:36.755580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:36.755580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:36.755717 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:36.755815 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:36.755993 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 13:42:36.756092 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 13:42:36.757570 23750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:36.759187 23750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:36.765604 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.761296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:35323 (local address 127.0.0.1:43378)
0924 13:42:36.761683 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:36.761691 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:36.761729 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:36.762191 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:36.762192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:36.762194 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:36.762236 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.762237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.763426 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.763427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.764304 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.764306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.765308 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.765309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.765388 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.765400 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 13:42:36.765467 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 13:42:36.765519 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260924 13:42:36.765628 24219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.761570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43378 (local address 127.19.53.194:35323)
0924 13:42:36.761758 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:36.761762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:36.761770 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:36.761806 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:36.761809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:36.761825 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:36.762175 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:36.762298 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.763331 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.764398 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.765159 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.765225 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.765275 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:36.765499 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 13:42:36.765542 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260924 13:42:36.766033 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.761296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:34969 (local address 127.0.0.1:42262)
0924 13:42:36.761460 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:36.761499 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:36.761574 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:36.761882 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:36.761885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:36.761887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:36.761970 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.761971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.763234 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.763235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.764802 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.764805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.765771 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.765772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.765835 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.765838 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:36.765899 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 13:42:36.765968 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":60,"threads_started":1}
I20260924 13:42:36.766088 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.761561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.19.53.193:34969)
0924 13:42:36.761709 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:36.761714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:36.761726 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:42:36.761769 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:36.761771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:36.761788 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:36.761879 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:36.762044 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.763147 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.764957 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.765601 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.765672 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.765734 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:36.765958 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 13:42:36.766004 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260924 13:42:36.766358 24007 tablet_service.cc:1467] Processing CreateTablet for tablet a24e48b87e854a7b8f9e93c6afc09a2c (DEFAULT_TABLE table=test-table [id=6a47e4bdbeaa454ea6a48adc36d26ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:36.766724 24007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24e48b87e854a7b8f9e93c6afc09a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:36.767252 24215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.761305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:44075 (local address 127.0.0.1:54126)
0924 13:42:36.761733 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:36.761742 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:36.761783 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:36.763259 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:36.763260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:36.763263 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:36.763308 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.763309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.764627 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.764628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.766042 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.766045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.767083 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.767084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.767128 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.767132 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:36.767179 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 13:42:36.767208 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":1,"thread_start_us":52,"threads_started":1}
I20260924 13:42:36.767311 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.761460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.19.53.195:44075)
0924 13:42:36.761860 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:36.761864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:36.761873 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:36.761936 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:36.761939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:36.761956 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:36.762048 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:36.763384 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.764529 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.766305 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.767004 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.767084 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.767139 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:36.767191 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 13:42:36.767232 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":108,"threads_started":1}
I20260924 13:42:36.767891 24140 tablet_service.cc:1467] Processing CreateTablet for tablet a24e48b87e854a7b8f9e93c6afc09a2c (DEFAULT_TABLE table=test-table [id=6a47e4bdbeaa454ea6a48adc36d26ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:36.768230 24140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24e48b87e854a7b8f9e93c6afc09a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:36.769618 24220 tablet_bootstrap.cc:492] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda: Bootstrap starting.
I20260924 13:42:36.770257 23874 tablet_service.cc:1467] Processing CreateTablet for tablet a24e48b87e854a7b8f9e93c6afc09a2c (DEFAULT_TABLE table=test-table [id=6a47e4bdbeaa454ea6a48adc36d26ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:36.770377 24220 tablet_bootstrap.cc:654] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:36.770561 23874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24e48b87e854a7b8f9e93c6afc09a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:36.770988 24220 log.cc:824] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:36.771394 24221 tablet_bootstrap.cc:492] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4: Bootstrap starting.
I20260924 13:42:36.771785 24220 tablet_bootstrap.cc:492] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda: No bootstrap required, opened a new log
I20260924 13:42:36.771853 24220 ts_tablet_manager.cc:1397] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:42:36.772107 24221 tablet_bootstrap.cc:654] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:36.772854 24221 log.cc:824] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:36.773458 24224 tablet_bootstrap.cc:492] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0: Bootstrap starting.
I20260924 13:42:36.773664 24221 tablet_bootstrap.cc:492] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4: No bootstrap required, opened a new log
I20260924 13:42:36.773739 24221 ts_tablet_manager.cc:1397] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:42:36.774150 24224 tablet_bootstrap.cc:654] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:36.774050 24220 raft_consensus.cc:348] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75683a37c4478d87265bd0ba9476b0" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34969 } } peers { permanent_uuid: "771374348bc04268acc1d684be0fadda" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 35323 } } peers { permanent_uuid: "4accd46f8a5f409fbfebe760f926d2f4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 44075 } }
I20260924 13:42:36.774214 24220 raft_consensus.cc:374] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:36.774255 24220 raft_consensus.cc:729] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771374348bc04268acc1d684be0fadda, State: Initialized, Role: FOLLOWER
I20260924 13:42:36.774382 24220 consensus_queue.cc:260] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75683a37c4478d87265bd0ba9476b0" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34969 } } peers { permanent_uuid: "771374348bc04268acc1d684be0fadda" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 35323 } } peers { permanent_uuid: "4accd46f8a5f409fbfebe760f926d2f4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 44075 } }
I20260924 13:42:36.774660 24220 ts_tablet_manager.cc:1428] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 13:42:36.774746 24224 log.cc:824] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:36.774806 24073 heartbeater.cc:503] Master 127.19.53.254:41079 was elected leader, sending a full tablet report...
I20260924 13:42:36.775556 24224 tablet_bootstrap.cc:492] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0: No bootstrap required, opened a new log
I20260924 13:42:36.775626 24224 ts_tablet_manager.cc:1397] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:42:36.775705 24221 raft_consensus.cc:348] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75683a37c4478d87265bd0ba9476b0" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34969 } } peers { permanent_uuid: "771374348bc04268acc1d684be0fadda" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 35323 } } peers { permanent_uuid: "4accd46f8a5f409fbfebe760f926d2f4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 44075 } }
I20260924 13:42:36.775878 24221 raft_consensus.cc:374] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:36.775918 24221 raft_consensus.cc:729] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4accd46f8a5f409fbfebe760f926d2f4, State: Initialized, Role: FOLLOWER
I20260924 13:42:36.776051 24221 consensus_queue.cc:260] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75683a37c4478d87265bd0ba9476b0" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34969 } } peers { permanent_uuid: "771374348bc04268acc1d684be0fadda" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 35323 } } peers { permanent_uuid: "4accd46f8a5f409fbfebe760f926d2f4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 44075 } }
I20260924 13:42:36.776326 24206 heartbeater.cc:503] Master 127.19.53.254:41079 was elected leader, sending a full tablet report...
I20260924 13:42:36.776551 24221 ts_tablet_manager.cc:1428] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:42:36.777632 24224 raft_consensus.cc:348] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75683a37c4478d87265bd0ba9476b0" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34969 } } peers { permanent_uuid: "771374348bc04268acc1d684be0fadda" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 35323 } } peers { permanent_uuid: "4accd46f8a5f409fbfebe760f926d2f4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 44075 } }
I20260924 13:42:36.777755 24224 raft_consensus.cc:374] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:36.777777 24224 raft_consensus.cc:729] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af75683a37c4478d87265bd0ba9476b0, State: Initialized, Role: FOLLOWER
I20260924 13:42:36.777880 24224 consensus_queue.cc:260] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75683a37c4478d87265bd0ba9476b0" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34969 } } peers { permanent_uuid: "771374348bc04268acc1d684be0fadda" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 35323 } } peers { permanent_uuid: "4accd46f8a5f409fbfebe760f926d2f4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 44075 } }
I20260924 13:42:36.778154 24224 ts_tablet_manager.cc:1428] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:42:36.778187 23940 heartbeater.cc:503] Master 127.19.53.254:41079 was elected leader, sending a full tablet report...
W20260924 13:42:36.832242 24074 tablet.cc:2369] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:42:36.936759 23941 tablet.cc:2369] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:42:36.958910 24227 raft_consensus.cc:482] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:36.959029 24227 raft_consensus.cc:504] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75683a37c4478d87265bd0ba9476b0" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34969 } } peers { permanent_uuid: "771374348bc04268acc1d684be0fadda" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 35323 } } peers { permanent_uuid: "4accd46f8a5f409fbfebe760f926d2f4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 44075 } }
I20260924 13:42:36.959431 24227 leader_election.cc:288] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af75683a37c4478d87265bd0ba9476b0 (127.19.53.193:34969), 771374348bc04268acc1d684be0fadda (127.19.53.194:35323)
I20260924 13:42:36.962182 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.959661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:39885 (local address 127.19.53.193:34969)
0924 13:42:36.959707 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:36.959709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:36.959719 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:42:36.959773 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:36.959773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:36.959790 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:36.959849 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:36.960037 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.960517 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.961432 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.961781 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.961861 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.961869 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:36.962082 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 13:42:36.962119 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:36.962181 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.959595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:34969 (local address 127.19.53.195:39885)
0924 13:42:36.959654 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:36.959666 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:36.959709 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:36.959869 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:36.959870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:36.959871 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:36.959940 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.959942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.960597 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.960598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.961323 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.961324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.961869 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.961870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.961997 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.962000 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:36.962073 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 13:42:36.962111 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 13:42:36.962325 24219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.959717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:59571 (local address 127.19.53.194:35323)
0924 13:42:36.959760 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:36.959762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:36.959868 (+ 106us) server_negotiation.cc:407] Connection header received
0924 13:42:36.959930 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:36.959931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:36.959940 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:36.960026 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:36.960197 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.960650 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.961558 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.961989 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.962068 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.962076 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:36.962228 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 13:42:36.962261 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:42:36.962340 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.959641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:35323 (local address 127.19.53.195:59571)
0924 13:42:36.959811 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:36.959818 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:36.959878 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:36.960030 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:36.960031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:36.960033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:36.960103 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.960105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.960732 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.960733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.961468 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:36.961470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.962069 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.962069 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:36.962149 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.962152 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:36.962222 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:42:36.962261 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260924 13:42:36.962553 23894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24e48b87e854a7b8f9e93c6afc09a2c" candidate_uuid: "4accd46f8a5f409fbfebe760f926d2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af75683a37c4478d87265bd0ba9476b0" is_pre_election: true
I20260924 13:42:36.962740 23894 raft_consensus.cc:2454] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4accd46f8a5f409fbfebe760f926d2f4 in term 0.
I20260924 13:42:36.963013 24094 leader_election.cc:302] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4accd46f8a5f409fbfebe760f926d2f4, af75683a37c4478d87265bd0ba9476b0; no voters:
I20260924 13:42:36.963006 24027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24e48b87e854a7b8f9e93c6afc09a2c" candidate_uuid: "4accd46f8a5f409fbfebe760f926d2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771374348bc04268acc1d684be0fadda" is_pre_election: true
I20260924 13:42:36.963160 24027 raft_consensus.cc:2454] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4accd46f8a5f409fbfebe760f926d2f4 in term 0.
I20260924 13:42:36.963158 24227 raft_consensus.cc:2781] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:36.963215 24227 raft_consensus.cc:482] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:36.963250 24227 raft_consensus.cc:3037] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:36.963938 24227 raft_consensus.cc:504] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75683a37c4478d87265bd0ba9476b0" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34969 } } peers { permanent_uuid: "771374348bc04268acc1d684be0fadda" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 35323 } } peers { permanent_uuid: "4accd46f8a5f409fbfebe760f926d2f4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 44075 } }
I20260924 13:42:36.964149 24227 leader_election.cc:288] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [CANDIDATE]: Term 1 election: Requested vote from peers af75683a37c4478d87265bd0ba9476b0 (127.19.53.193:34969), 771374348bc04268acc1d684be0fadda (127.19.53.194:35323)
I20260924 13:42:36.964253 23894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24e48b87e854a7b8f9e93c6afc09a2c" candidate_uuid: "4accd46f8a5f409fbfebe760f926d2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af75683a37c4478d87265bd0ba9476b0"
I20260924 13:42:36.964287 24027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24e48b87e854a7b8f9e93c6afc09a2c" candidate_uuid: "4accd46f8a5f409fbfebe760f926d2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771374348bc04268acc1d684be0fadda"
I20260924 13:42:36.964344 23894 raft_consensus.cc:3037] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:36.964377 24027 raft_consensus.cc:3037] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:36.965003 23894 raft_consensus.cc:2454] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4accd46f8a5f409fbfebe760f926d2f4 in term 1.
I20260924 13:42:36.965150 24027 raft_consensus.cc:2454] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4accd46f8a5f409fbfebe760f926d2f4 in term 1.
I20260924 13:42:36.965186 24094 leader_election.cc:302] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4accd46f8a5f409fbfebe760f926d2f4, af75683a37c4478d87265bd0ba9476b0; no voters:
I20260924 13:42:36.965281 24227 raft_consensus.cc:2781] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:36.965466 24227 raft_consensus.cc:686] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [term 1 LEADER]: Becoming Leader. State: Replica: 4accd46f8a5f409fbfebe760f926d2f4, State: Running, Role: LEADER
I20260924 13:42:36.965571 24227 consensus_queue.cc:237] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75683a37c4478d87265bd0ba9476b0" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34969 } } peers { permanent_uuid: "771374348bc04268acc1d684be0fadda" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 35323 } } peers { permanent_uuid: "4accd46f8a5f409fbfebe760f926d2f4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 44075 } }
I20260924 13:42:36.966440 23750 catalog_manager.cc:5495] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4accd46f8a5f409fbfebe760f926d2f4 (127.19.53.195). New cstate: current_term: 1 leader_uuid: "4accd46f8a5f409fbfebe760f926d2f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75683a37c4478d87265bd0ba9476b0" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771374348bc04268acc1d684be0fadda" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 35323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4accd46f8a5f409fbfebe760f926d2f4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 44075 } health_report { overall_health: HEALTHY } } }
W20260924 13:42:36.984339 24207 tablet.cc:2369] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:42:36.996106 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:36.992375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.19.53.195:44075)
0924 13:42:36.992437 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:36.992439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:36.992447 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:36.992486 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:36.992487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:36.992509 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:36.992724 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:36.992900 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.993634 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.995100 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:36.995426 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:36.995492 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:36.995707 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:36.995854 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:36.995948 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:36.995968 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 13:42:36.995993 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 13:42:36.998042 24027 raft_consensus.cc:1261] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda [term 1 FOLLOWER]: Refusing update from remote peer 4accd46f8a5f409fbfebe760f926d2f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:36.998410 24227 consensus_queue.cc:1035] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "771374348bc04268acc1d684be0fadda" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:36.998590 23894 raft_consensus.cc:1261] T a24e48b87e854a7b8f9e93c6afc09a2c P af75683a37c4478d87265bd0ba9476b0 [term 1 FOLLOWER]: Refusing update from remote peer 4accd46f8a5f409fbfebe760f926d2f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:36.998852 24227 consensus_queue.cc:1035] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af75683a37c4478d87265bd0ba9476b0" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:37.001096 24234 mvcc.cc:205] Tried to move back new op lower bound from 7332894134260973568 to 7332894134131724288. Current Snapshot: MvccSnapshot[applied={T|T < 7332894134260973568}]
I20260924 13:42:37.004426 23749 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:37.004541 23749 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:37.005261 23749 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2893176d2b454a28b076359a806a31c1: Sending DeleteTablet for 3 replicas of tablet a24e48b87e854a7b8f9e93c6afc09a2c
I20260924 13:42:37.005630 24007 tablet_service.cc:1514] Processing DeleteTablet for tablet a24e48b87e854a7b8f9e93c6afc09a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:36 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:43378
I20260924 13:42:37.005667 24140 tablet_service.cc:1514] Processing DeleteTablet for tablet a24e48b87e854a7b8f9e93c6afc09a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:36 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:54126
I20260924 13:42:37.005654 23874 tablet_service.cc:1514] Processing DeleteTablet for tablet a24e48b87e854a7b8f9e93c6afc09a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:36 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:42262
I20260924 13:42:37.005856 24243 tablet_replica.cc:330] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda: stopping tablet replica
I20260924 13:42:37.005995 24243 raft_consensus.cc:2229] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:37.006114 24243 raft_consensus.cc:2258] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:37.006115 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 23812
I20260924 13:42:37.006248 24244 tablet_replica.cc:330] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4: stopping tablet replica
I20260924 13:42:37.006322 24244 raft_consensus.cc:2229] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:42:37.006404 24244 raft_consensus.cc:2258] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:37.007192 24244 ts_tablet_manager.cc:1905] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:37.008800 24244 ts_tablet_manager.cc:1918] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:37.008874 24244 log.cc:1196] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/a24e48b87e854a7b8f9e93c6afc09a2c
I20260924 13:42:37.009148 24244 ts_tablet_manager.cc:1939] T a24e48b87e854a7b8f9e93c6afc09a2c P 4accd46f8a5f409fbfebe760f926d2f4: Deleting consensus metadata
I20260924 13:42:37.009553 23736 catalog_manager.cc:4841] TS 4accd46f8a5f409fbfebe760f926d2f4 (127.19.53.195:44075): tablet a24e48b87e854a7b8f9e93c6afc09a2c (table test-table [id=6a47e4bdbeaa454ea6a48adc36d26ef5]) successfully deleted
I20260924 13:42:37.009661 24243 ts_tablet_manager.cc:1905] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:37.011204 24243 ts_tablet_manager.cc:1918] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:37.011281 24243 log.cc:1196] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/a24e48b87e854a7b8f9e93c6afc09a2c
I20260924 13:42:37.011550 24243 ts_tablet_manager.cc:1939] T a24e48b87e854a7b8f9e93c6afc09a2c P 771374348bc04268acc1d684be0fadda: Deleting consensus metadata
I20260924 13:42:37.011951 23734 catalog_manager.cc:4841] TS 771374348bc04268acc1d684be0fadda (127.19.53.194:35323): tablet a24e48b87e854a7b8f9e93c6afc09a2c (table test-table [id=6a47e4bdbeaa454ea6a48adc36d26ef5]) successfully deleted
W20260924 13:42:37.014647 23735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:34969 (error 108)
I20260924 13:42:37.014894 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 23945
I20260924 13:42:37.015502 24215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.014980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:34969 (local address 127.0.0.1:42274)
0924 13:42:37.015409 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:37.015453 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:34969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404}
W20260924 13:42:37.015622 23735 catalog_manager.cc:4584] TS af75683a37c4478d87265bd0ba9476b0 (127.19.53.193:34969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a24e48b87e854a7b8f9e93c6afc09a2c: Network error: Client connection negotiation failed: client connection to 127.19.53.193:34969: connect: Connection refused (error 111)
I20260924 13:42:37.020880 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 24078
I20260924 13:42:37.027384 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 23719
2026-09-24T13:42:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2066 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T13:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:37Z Disabled control of system clock
I20260924 13:42:37.057725 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:35089
--webserver_interface=127.19.53.254
--webserver_port=0
--builtin_ntp_servers=127.19.53.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:35089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 13:42:37.142104 24250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:37.142334 24250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:37.142367 24250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:37.143483 24250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:37.143548 24250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:37.143568 24250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:37.143587 24250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:37.143620 24250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:37.144933 24250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:35089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:35089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:37.145200 24250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:37.145483 24250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:37.150134 24256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:37.150103 24258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:37.150113 24255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:37.150295 24250 server_base.cc:1034] running on GCE node
I20260924 13:42:37.150563 24250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:37.150861 24250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:37.152031 24250 hybrid_clock.cc:648] HybridClock initialized: now 1790257357152012 us; error 34 us; skew 500 ppm
I20260924 13:42:37.154175 24250 webserver.cc:460] Webserver started at http://127.19.53.254:45867/ using document root <none> and password file <none>
I20260924 13:42:37.154415 24250 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:37.154481 24250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:37.154608 24250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:37.155579 24250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "28bcb702cffd4eeab6636b7eb1f1dcfd"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "d5766520090ae04a892fde458fa3a620"
server_key_iv: "a9a70e6f7048c655d92091bf835851d2"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.155927 24250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "28bcb702cffd4eeab6636b7eb1f1dcfd"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "d5766520090ae04a892fde458fa3a620"
server_key_iv: "a9a70e6f7048c655d92091bf835851d2"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.157148 24250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:37.157817 24264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:37.158003 24250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:37.158100 24250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "28bcb702cffd4eeab6636b7eb1f1dcfd"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "d5766520090ae04a892fde458fa3a620"
server_key_iv: "a9a70e6f7048c655d92091bf835851d2"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.158183 24250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:37.161310 24250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:37.161577 24250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:37.161715 24250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:37.165380 24250 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:35089
I20260924 13:42:37.165410 24316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:35089 every 8 connection(s)
I20260924 13:42:37.165727 24250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:37.166378 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:37.168962 24317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd: Bootstrap starting.
I20260924 13:42:37.169492 24317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:37.169718 24317 log.cc:824] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:37.170392 24317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd: No bootstrap required, opened a new log
I20260924 13:42:37.171576 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 24250
I20260924 13:42:37.171675 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:37.171851 19671 external_mini_cluster.cc:1428] Setting key ff5c4f0a2320ca60a305f46fa5898c0a
I20260924 13:42:37.171980 24317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bcb702cffd4eeab6636b7eb1f1dcfd" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35089 } }
I20260924 13:42:37.172169 24317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:37.172211 24317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bcb702cffd4eeab6636b7eb1f1dcfd, State: Initialized, Role: FOLLOWER
I20260924 13:42:37.172359 24317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bcb702cffd4eeab6636b7eb1f1dcfd" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35089 } }
I20260924 13:42:37.172437 24317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:37.172488 24317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:37.172541 24317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:37.173022 24317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bcb702cffd4eeab6636b7eb1f1dcfd" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35089 } }
I20260924 13:42:37.173102 24317 leader_election.cc:302] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28bcb702cffd4eeab6636b7eb1f1dcfd; no voters:
I20260924 13:42:37.173220 24317 leader_election.cc:288] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:37.173280 24322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:37.173411 24322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [term 1 LEADER]: Becoming Leader. State: Replica: 28bcb702cffd4eeab6636b7eb1f1dcfd, State: Running, Role: LEADER
I20260924 13:42:37.173451 24317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:37.173497 24322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bcb702cffd4eeab6636b7eb1f1dcfd" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35089 } }
I20260924 13:42:37.173954 24323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28bcb702cffd4eeab6636b7eb1f1dcfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bcb702cffd4eeab6636b7eb1f1dcfd" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35089 } } }
I20260924 13:42:37.174126 24323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:37.174183 24324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28bcb702cffd4eeab6636b7eb1f1dcfd. Latest consensus state: current_term: 1 leader_uuid: "28bcb702cffd4eeab6636b7eb1f1dcfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bcb702cffd4eeab6636b7eb1f1dcfd" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35089 } } }
I20260924 13:42:37.174266 24324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:37.174544 24331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:37.175071 24331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:37.176857 24331 catalog_manager.cc:1347] Generated new cluster ID: cf7538647b39459fbaff5f837fdfd776
I20260924 13:42:37.176914 24331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:37.182793 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.172348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38364 (local address 127.19.53.254:35089)
0924 13:42:37.172588 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:37.172597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:37.174978 (+ 2381us) server_negotiation.cc:407] Connection header received
0924 13:42:37.175145 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:37.175154 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:37.175199 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:37.175338 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:37.182339 (+ 7001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:37.182342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:37.182343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:37.182348 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:37.182394 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:37.182406 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:37.182489 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:37.182556 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 13:42:37.182614 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260924 13:42:37.193396 24331 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:37.193573 24331 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:37.198838 24331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd: Generated new TSK 0
I20260924 13:42:37.199024 24331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:42:37.203624 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:35089
--builtin_ntp_servers=127.19.53.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 13:42:37.285723 24341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:37.285991 24341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:37.286060 24341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:37.287246 24341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:37.287312 24341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:37.287360 24341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:37.288734 24341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:35089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:37.288995 24341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:37.289278 24341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:37.295091 24347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:37.295094 24346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:37.295440 24349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:37.296096 24341 server_base.cc:1034] running on GCE node
I20260924 13:42:37.296351 24341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:37.296648 24341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:37.297825 24341 hybrid_clock.cc:648] HybridClock initialized: now 1790257357297816 us; error 47 us; skew 500 ppm
I20260924 13:42:37.299831 24341 webserver.cc:460] Webserver started at http://127.19.53.193:38115/ using document root <none> and password file <none>
I20260924 13:42:37.300079 24341 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:37.300133 24341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:37.300256 24341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:37.301609 24341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "8b4e795f390d475e9735eeb5c1d0fd5f"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "236444a00f57cddda743f82f5a137f5e"
server_key_iv: "3de2de4328563ac3c8454b93c66fa30f"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.302012 24341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "8b4e795f390d475e9735eeb5c1d0fd5f"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "236444a00f57cddda743f82f5a137f5e"
server_key_iv: "3de2de4328563ac3c8454b93c66fa30f"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.303625 24341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:37.304549 24355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:37.304742 24341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:37.304847 24341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "8b4e795f390d475e9735eeb5c1d0fd5f"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "236444a00f57cddda743f82f5a137f5e"
server_key_iv: "3de2de4328563ac3c8454b93c66fa30f"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.304926 24341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:37.308300 24341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:37.308581 24341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:37.308707 24341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:37.309082 24341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:37.309404 24341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:37.309438 24341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:37.309492 24341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:37.309520 24341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:37.314936 24341 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:35217
I20260924 13:42:37.315008 24468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:35217 every 8 connection(s)
I20260924 13:42:37.315320 24341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:42:37.318388 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 24341
I20260924 13:42:37.318511 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:42:37.318719 19671 external_mini_cluster.cc:1428] Setting key 094e6e8a257de7f78d69d20570395574
I20260924 13:42:37.319247 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.316177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:42557 (local address 127.19.53.254:35089)
0924 13:42:37.316281 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:37.316288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:37.316512 (+ 224us) server_negotiation.cc:407] Connection header received
0924 13:42:37.317028 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:37.317030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:37.317036 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:37.317088 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:37.318983 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:37.318985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:37.318985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:37.318989 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:37.319005 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:37.319009 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:37.319087 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:37.319163 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 13:42:37.319178 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 13:42:37.319450 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.316147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:35089 (local address 127.19.53.193:42557)
0924 13:42:37.316372 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:37.316404 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:37.316919 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:37.317168 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:37.317172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:37.317188 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:37.318814 (+ 1626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:42:37.318814 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:37.318851 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:42:37.318885 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:37.319084 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:37.319090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:37.319093 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:37.319173 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:42:37.319231 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260924 13:42:37.319669 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:35089
--builtin_ntp_servers=127.19.53.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 13:42:37.319934 24469 heartbeater.cc:347] Connected to a master server at 127.19.53.254:35089
I20260924 13:42:37.320019 24469 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:37.320181 24469 heartbeater.cc:511] Master 127.19.53.254:35089 requested a full tablet report, sending...
I20260924 13:42:37.320710 24281 ts_manager.cc:194] Registered new tserver with Master: 8b4e795f390d475e9735eeb5c1d0fd5f (127.19.53.193:35217)
W20260924 13:42:37.398875 24472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:37.399067 24472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:37.399089 24472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:37.400152 24472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:37.400180 24472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:37.400204 24472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:37.401422 24472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:35089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:37.401600 24472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:37.401829 24472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:37.406486 24478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:37.406502 24477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:37.406694 24472 server_base.cc:1034] running on GCE node
W20260924 13:42:37.406510 24480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:37.407030 24472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:37.407274 24472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:37.408469 24472 hybrid_clock.cc:648] HybridClock initialized: now 1790257357408459 us; error 33 us; skew 500 ppm
I20260924 13:42:37.410138 24472 webserver.cc:460] Webserver started at http://127.19.53.194:33231/ using document root <none> and password file <none>
I20260924 13:42:37.410365 24472 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:37.410408 24472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:37.410558 24472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:37.411569 24472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "09b3579877a0e2effccca1a691b38fd4"
server_key_iv: "07edb98c9850f5af524d61c211cefed5"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.411914 24472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "09b3579877a0e2effccca1a691b38fd4"
server_key_iv: "07edb98c9850f5af524d61c211cefed5"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.413280 24472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:37.414048 24486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:37.414283 24472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:37.414377 24472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "09b3579877a0e2effccca1a691b38fd4"
server_key_iv: "07edb98c9850f5af524d61c211cefed5"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.414462 24472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:37.417644 24472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:37.417901 24472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:37.418063 24472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:37.418267 24472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:37.418560 24472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:37.418612 24472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:37.418649 24472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:37.418676 24472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:37.424407 24472 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:46483
I20260924 13:42:37.424472 24599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:46483 every 8 connection(s)
I20260924 13:42:37.424773 24472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:42:37.428459 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.425482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:41477 (local address 127.19.53.254:35089)
0924 13:42:37.425530 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:37.425532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:37.425775 (+ 243us) server_negotiation.cc:407] Connection header received
0924 13:42:37.426281 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:37.426282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:37.426288 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:37.426338 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:37.428287 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:37.428288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:37.428288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:37.428291 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:37.428303 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:37.428306 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:37.428352 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:37.428396 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 13:42:37.428408 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:42:37.428656 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.425481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:35089 (local address 127.19.53.194:41477)
0924 13:42:37.425652 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:37.425682 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:37.426186 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:37.426388 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:37.426391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:37.426398 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:37.428159 (+ 1761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:42:37.428160 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:37.428187 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:42:37.428213 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:37.428349 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:37.428352 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:37.428353 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:42:37.428402 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 13:42:37.428445 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260924 13:42:37.429088 24600 heartbeater.cc:347] Connected to a master server at 127.19.53.254:35089
I20260924 13:42:37.429164 24600 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:37.429266 24600 heartbeater.cc:511] Master 127.19.53.254:35089 requested a full tablet report, sending...
I20260924 13:42:37.429558 24281 ts_manager.cc:194] Registered new tserver with Master: 2defa8306a5d4f1cb1e1b1d1de255b3e (127.19.53.194:46483)
I20260924 13:42:37.434481 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 24472
I20260924 13:42:37.434584 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:42:37.434767 19671 external_mini_cluster.cc:1428] Setting key 23997db25d8ac8c5d6e68b8cbb99a5fe
I20260924 13:42:37.435643 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:35089
--builtin_ntp_servers=127.19.53.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 13:42:37.515383 24603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:37.515580 24603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:37.515601 24603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:37.516775 24603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:37.516815 24603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:37.516865 24603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:37.518221 24603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:35089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:37.518463 24603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:37.518772 24603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:37.523579 24608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:37.523600 24609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:37.523697 24603 server_base.cc:1034] running on GCE node
W20260924 13:42:37.523581 24611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:37.524010 24603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:37.524290 24603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:37.525457 24603 hybrid_clock.cc:648] HybridClock initialized: now 1790257357525429 us; error 44 us; skew 500 ppm
I20260924 13:42:37.527616 24603 webserver.cc:460] Webserver started at http://127.19.53.195:33369/ using document root <none> and password file <none>
I20260924 13:42:37.527856 24603 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:37.527925 24603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:37.528059 24603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:37.529093 24603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "8d2141d47d08d2241af0d259924ccfef"
server_key_iv: "ee459d2d8672fec8f3786296f0d012e6"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.529451 24603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "8d2141d47d08d2241af0d259924ccfef"
server_key_iv: "ee459d2d8672fec8f3786296f0d012e6"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.530802 24603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:37.531474 24617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:37.531631 24603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:37.531721 24603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "8d2141d47d08d2241af0d259924ccfef"
server_key_iv: "ee459d2d8672fec8f3786296f0d012e6"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.531821 24603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:37.535046 24603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:37.535295 24603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:37.535430 24603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:37.535629 24603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:37.535914 24603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:37.535967 24603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:37.536005 24603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:37.536031 24603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:37.541608 24603 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:34007
I20260924 13:42:37.541673 24730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:34007 every 8 connection(s)
I20260924 13:42:37.542058 24603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:42:37.545713 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.542766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:38995 (local address 127.19.53.254:35089)
0924 13:42:37.542822 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:37.542824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:37.543029 (+ 205us) server_negotiation.cc:407] Connection header received
0924 13:42:37.543528 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:37.543531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:37.543538 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:37.543589 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:37.545412 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:37.545413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:37.545414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:37.545418 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:37.545435 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:37.545485 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:37.545555 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:37.545626 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 13:42:37.545642 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:42:37.545933 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.542732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:35089 (local address 127.19.53.195:38995)
0924 13:42:37.542903 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:37.542935 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:37.543404 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:37.543668 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:37.543672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:37.543681 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:37.545243 (+ 1562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:42:37.545244 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:37.545271 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:42:37.545298 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:37.545551 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:37.545557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:37.545561 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:37.545637 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 13:42:37.545699 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260924 13:42:37.546394 24731 heartbeater.cc:347] Connected to a master server at 127.19.53.254:35089
I20260924 13:42:37.546470 24731 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:37.546636 24731 heartbeater.cc:511] Master 127.19.53.254:35089 requested a full tablet report, sending...
I20260924 13:42:37.547042 24281 ts_manager.cc:194] Registered new tserver with Master: 3e7c5ff888dd4618ac4808ea7e19a2b3 (127.19.53.195:34007)
I20260924 13:42:37.550066 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 24603
I20260924 13:42:37.550154 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:42:37.550298 19671 external_mini_cluster.cc:1428] Setting key a70b6bfe5722f80e30daf873b866e5c5
I20260924 13:42:37.551213 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:42:37.554145 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.552567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.19.53.254:35089)
0924 13:42:37.552625 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:37.552627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:37.552757 (+ 130us) server_negotiation.cc:407] Connection header received
0924 13:42:37.552806 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:37.552807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:37.552812 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:37.552857 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:37.553844 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:37.553846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:37.553846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:37.553850 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:37.553863 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:37.553866 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:37.553963 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:37.554027 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 13:42:37.554038 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 13:42:37.556313 24281 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:37.557886 24281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:37.563072 24739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.560142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:35217 (local address 127.0.0.1:37048)
0924 13:42:37.560329 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:37.560370 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:37.560427 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:37.560894 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:37.560901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:37.560908 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:37.562552 (+ 1644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:42:37.562553 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:37.562571 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:42:37.562598 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:37.562878 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:37.562885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:37.562889 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:37.562965 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 13:42:37.562978 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260924 13:42:37.563383 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.560283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.19.53.193:35217)
0924 13:42:37.560503 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:37.560508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:37.560517 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:37.560543 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:37.560547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:37.560559 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:37.560897 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:37.562698 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:37.562706 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:37.562707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:37.562711 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:37.562748 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:37.562775 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:37.562882 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:37.563283 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 13:42:37.563300 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":93,"threads_started":1}
I20260924 13:42:37.563526 24740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.560489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:46483 (local address 127.0.0.1:48736)
0924 13:42:37.560681 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:37.560691 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:37.560734 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:37.561040 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:37.561041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:37.561044 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:37.563111 (+ 2067us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:42:37.563112 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:37.563117 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:42:37.563127 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:37.563382 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:37.563392 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:37.563392 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 13:42:37.563429 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 13:42:37.563443 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
I20260924 13:42:37.563532 24743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.560230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:34007 (local address 127.0.0.1:47050)
0924 13:42:37.560639 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:37.560649 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:37.560691 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:37.561314 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:37.561315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:37.561317 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:37.562931 (+ 1614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:42:37.562932 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:37.562940 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:42:37.562953 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:37.563415 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:37.563415 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:37.563416 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:42:37.563462 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 13:42:37.563473 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":92,"threads_started":1}
I20260924 13:42:37.563861 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.560383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.19.53.195:34007)
0924 13:42:37.560894 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:37.560898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:37.560907 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:37.560931 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:37.560933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:37.560942 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:37.561018 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:37.563027 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:37.563071 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:37.563072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:37.563076 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:37.563112 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:37.563353 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:37.563422 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:37.563789 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 13:42:37.563803 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":80,"threads_started":1}
I20260924 13:42:37.563877 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.560643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.19.53.194:46483)
0924 13:42:37.560855 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:37.560871 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:37.560882 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:42:37.560912 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:37.560914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:37.560925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:37.561525 (+ 600us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:37.563216 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:37.563223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:37.563224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:37.563229 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:37.563273 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:37.563297 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:37.563747 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:37.563774 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 13:42:37.563791 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":108,"threads_started":1}
I20260924 13:42:37.564102 24403 tablet_service.cc:1467] Processing CreateTablet for tablet ca8410df229846ce82a500ebdf239a75 (DEFAULT_TABLE table=test-table [id=64ae8fc48ebb40319814a63ec5979d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:37.564522 24403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8410df229846ce82a500ebdf239a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:37.564494 24665 tablet_service.cc:1467] Processing CreateTablet for tablet ca8410df229846ce82a500ebdf239a75 (DEFAULT_TABLE table=test-table [id=64ae8fc48ebb40319814a63ec5979d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:37.564559 24534 tablet_service.cc:1467] Processing CreateTablet for tablet ca8410df229846ce82a500ebdf239a75 (DEFAULT_TABLE table=test-table [id=64ae8fc48ebb40319814a63ec5979d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:37.564839 24534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8410df229846ce82a500ebdf239a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:37.564839 24665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8410df229846ce82a500ebdf239a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:37.567553 24745 tablet_bootstrap.cc:492] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f: Bootstrap starting.
I20260924 13:42:37.568068 24747 tablet_bootstrap.cc:492] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3: Bootstrap starting.
I20260924 13:42:37.568418 24745 tablet_bootstrap.cc:654] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:37.568763 24747 tablet_bootstrap.cc:654] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:37.568763 24745 log.cc:824] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:37.569100 24747 log.cc:824] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:37.569432 24746 tablet_bootstrap.cc:492] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e: Bootstrap starting.
I20260924 13:42:37.569621 24745 tablet_bootstrap.cc:492] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f: No bootstrap required, opened a new log
I20260924 13:42:37.569707 24745 ts_tablet_manager.cc:1397] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:37.569851 24747 tablet_bootstrap.cc:492] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3: No bootstrap required, opened a new log
I20260924 13:42:37.569960 24747 ts_tablet_manager.cc:1397] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:42:37.570137 24746 tablet_bootstrap.cc:654] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:37.570464 24746 log.cc:824] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:37.571245 24746 tablet_bootstrap.cc:492] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e: No bootstrap required, opened a new log
I20260924 13:42:37.571326 24746 ts_tablet_manager.cc:1397] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:37.572278 24747 raft_consensus.cc:348] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4e795f390d475e9735eeb5c1d0fd5f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35217 } } peers { permanent_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34007 } } peers { permanent_uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46483 } }
I20260924 13:42:37.572429 24747 raft_consensus.cc:374] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:37.572481 24747 raft_consensus.cc:729] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7c5ff888dd4618ac4808ea7e19a2b3, State: Initialized, Role: FOLLOWER
I20260924 13:42:37.572598 24747 consensus_queue.cc:260] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4e795f390d475e9735eeb5c1d0fd5f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35217 } } peers { permanent_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34007 } } peers { permanent_uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46483 } }
I20260924 13:42:37.572842 24747 ts_tablet_manager.cc:1428] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:42:37.572767 24745 raft_consensus.cc:348] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4e795f390d475e9735eeb5c1d0fd5f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35217 } } peers { permanent_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34007 } } peers { permanent_uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46483 } }
I20260924 13:42:37.572996 24731 heartbeater.cc:503] Master 127.19.53.254:35089 was elected leader, sending a full tablet report...
I20260924 13:42:37.573362 24745 raft_consensus.cc:374] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:37.573437 24745 raft_consensus.cc:729] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4e795f390d475e9735eeb5c1d0fd5f, State: Initialized, Role: FOLLOWER
I20260924 13:42:37.573599 24745 consensus_queue.cc:260] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4e795f390d475e9735eeb5c1d0fd5f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35217 } } peers { permanent_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34007 } } peers { permanent_uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46483 } }
I20260924 13:42:37.573876 24745 ts_tablet_manager.cc:1428] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 13:42:37.573892 24469 heartbeater.cc:503] Master 127.19.53.254:35089 was elected leader, sending a full tablet report...
I20260924 13:42:37.574297 24746 raft_consensus.cc:348] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4e795f390d475e9735eeb5c1d0fd5f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35217 } } peers { permanent_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34007 } } peers { permanent_uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46483 } }
I20260924 13:42:37.574522 24746 raft_consensus.cc:374] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:37.574581 24746 raft_consensus.cc:729] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2defa8306a5d4f1cb1e1b1d1de255b3e, State: Initialized, Role: FOLLOWER
I20260924 13:42:37.574702 24746 consensus_queue.cc:260] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4e795f390d475e9735eeb5c1d0fd5f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35217 } } peers { permanent_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34007 } } peers { permanent_uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46483 } }
I20260924 13:42:37.575018 24746 ts_tablet_manager.cc:1428] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 13:42:37.575088 24600 heartbeater.cc:503] Master 127.19.53.254:35089 was elected leader, sending a full tablet report...
I20260924 13:42:37.620906 24751 raft_consensus.cc:482] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:37.621079 24751 raft_consensus.cc:504] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4e795f390d475e9735eeb5c1d0fd5f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35217 } } peers { permanent_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34007 } } peers { permanent_uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46483 } }
I20260924 13:42:37.621451 24751 leader_election.cc:288] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b4e795f390d475e9735eeb5c1d0fd5f (127.19.53.193:35217), 2defa8306a5d4f1cb1e1b1d1de255b3e (127.19.53.194:46483)
I20260924 13:42:37.623594 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.621654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:46483 (local address 127.19.53.195:44325)
0924 13:42:37.621719 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:37.621732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:37.621772 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:37.621945 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:37.621946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:37.621949 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:37.623276 (+ 1327us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:42:37.623277 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:37.623286 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:42:37.623301 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:37.623459 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:37.623460 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:37.623461 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:42:37.623514 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 13:42:37.623529 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 13:42:37.623594 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.621726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:44325 (local address 127.19.53.194:46483)
0924 13:42:37.621774 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:37.621776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:37.621817 (+ 41us) server_negotiation.cc:407] Connection header received
0924 13:42:37.621848 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:37.621849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:37.621853 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:37.621917 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:37.623378 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:37.623380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:37.623381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:37.623385 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:37.623401 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:37.623406 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:37.623453 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:37.623515 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 13:42:37.623529 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:42:37.623811 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.621742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:54053 (local address 127.19.53.193:35217)
0924 13:42:37.621786 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:37.621789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:37.621932 (+ 143us) server_negotiation.cc:407] Connection header received
0924 13:42:37.622015 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:37.622016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:37.622022 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:37.622090 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:37.623569 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:37.623570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:37.623570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:37.623575 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:37.623590 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:37.623594 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:37.623641 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:37.623708 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 13:42:37.623725 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:42:37.623818 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.621655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:35217 (local address 127.19.53.195:54053)
0924 13:42:37.621850 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:37.621858 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:37.621947 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:37.622094 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:37.622095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:37.622110 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:37.623457 (+ 1347us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:42:37.623458 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:37.623463 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:42:37.623481 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:37.623647 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:37.623648 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:37.623649 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:42:37.623702 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 13:42:37.623715 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":23,"thread_start_us":130,"threads_started":1}
I20260924 13:42:37.623911 24554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8410df229846ce82a500ebdf239a75" candidate_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e" is_pre_election: true
I20260924 13:42:37.624069 24554 raft_consensus.cc:2454] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e7c5ff888dd4618ac4808ea7e19a2b3 in term 0.
I20260924 13:42:37.624282 24620 leader_election.cc:302] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2defa8306a5d4f1cb1e1b1d1de255b3e, 3e7c5ff888dd4618ac4808ea7e19a2b3; no voters:
I20260924 13:42:37.624444 24751 raft_consensus.cc:2781] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:37.624460 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8410df229846ce82a500ebdf239a75" candidate_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4e795f390d475e9735eeb5c1d0fd5f" is_pre_election: true
I20260924 13:42:37.624522 24751 raft_consensus.cc:482] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:37.624554 24751 raft_consensus.cc:3037] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:37.624603 24423 raft_consensus.cc:2454] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e7c5ff888dd4618ac4808ea7e19a2b3 in term 0.
I20260924 13:42:37.625305 24751 raft_consensus.cc:504] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4e795f390d475e9735eeb5c1d0fd5f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35217 } } peers { permanent_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34007 } } peers { permanent_uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46483 } }
I20260924 13:42:37.625447 24751 leader_election.cc:288] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [CANDIDATE]: Term 1 election: Requested vote from peers 8b4e795f390d475e9735eeb5c1d0fd5f (127.19.53.193:35217), 2defa8306a5d4f1cb1e1b1d1de255b3e (127.19.53.194:46483)
I20260924 13:42:37.625592 24554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8410df229846ce82a500ebdf239a75" candidate_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e"
I20260924 13:42:37.625603 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8410df229846ce82a500ebdf239a75" candidate_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4e795f390d475e9735eeb5c1d0fd5f"
I20260924 13:42:37.625665 24423 raft_consensus.cc:3037] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:37.625687 24554 raft_consensus.cc:3037] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:37.626386 24554 raft_consensus.cc:2454] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e7c5ff888dd4618ac4808ea7e19a2b3 in term 1.
I20260924 13:42:37.626390 24423 raft_consensus.cc:2454] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e7c5ff888dd4618ac4808ea7e19a2b3 in term 1.
I20260924 13:42:37.626554 24620 leader_election.cc:302] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2defa8306a5d4f1cb1e1b1d1de255b3e, 3e7c5ff888dd4618ac4808ea7e19a2b3; no voters:
I20260924 13:42:37.626667 24751 raft_consensus.cc:2781] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:37.626812 24751 raft_consensus.cc:686] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3e7c5ff888dd4618ac4808ea7e19a2b3, State: Running, Role: LEADER
I20260924 13:42:37.626902 24751 consensus_queue.cc:237] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4e795f390d475e9735eeb5c1d0fd5f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35217 } } peers { permanent_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34007 } } peers { permanent_uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46483 } }
I20260924 13:42:37.627740 24281 catalog_manager.cc:5495] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e7c5ff888dd4618ac4808ea7e19a2b3 (127.19.53.195). New cstate: current_term: 1 leader_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4e795f390d475e9735eeb5c1d0fd5f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e7c5ff888dd4618ac4808ea7e19a2b3" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46483 } health_report { overall_health: UNKNOWN } } }
I20260924 13:42:37.636168 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.634447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.19.53.195:34007)
0924 13:42:37.634505 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:37.634506 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:37.634514 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:37.634579 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:37.634579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:37.634588 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:37.634632 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:37.635941 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:37.635942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:37.635942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:37.635945 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:37.635957 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:37.635960 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:37.636006 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:37.636059 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 13:42:37.636086 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:42:37.637818 24423 raft_consensus.cc:1261] T ca8410df229846ce82a500ebdf239a75 P 8b4e795f390d475e9735eeb5c1d0fd5f [term 1 FOLLOWER]: Refusing update from remote peer 3e7c5ff888dd4618ac4808ea7e19a2b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:37.637882 24554 raft_consensus.cc:1261] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e [term 1 FOLLOWER]: Refusing update from remote peer 3e7c5ff888dd4618ac4808ea7e19a2b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:37.638161 24751 consensus_queue.cc:1035] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b4e795f390d475e9735eeb5c1d0fd5f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:37.638290 24755 consensus_queue.cc:1035] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2defa8306a5d4f1cb1e1b1d1de255b3e" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:37.642779 24279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:37.642884 24279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:37.643615 24279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28bcb702cffd4eeab6636b7eb1f1dcfd: Sending DeleteTablet for 3 replicas of tablet ca8410df229846ce82a500ebdf239a75
I20260924 13:42:37.643949 24403 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8410df229846ce82a500ebdf239a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:37 UTC) from {username='slave'} at 127.0.0.1:37048
I20260924 13:42:37.643944 24665 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8410df229846ce82a500ebdf239a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:37 UTC) from {username='slave'} at 127.0.0.1:47050
I20260924 13:42:37.644155 24534 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8410df229846ce82a500ebdf239a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:37 UTC) from {username='slave'} at 127.0.0.1:48736
I20260924 13:42:37.644361 24770 tablet_replica.cc:330] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e: stopping tablet replica
I20260924 13:42:37.644464 24770 raft_consensus.cc:2229] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:37.644569 24770 raft_consensus.cc:2258] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:37.644673 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 24341
I20260924 13:42:37.645522 24770 ts_tablet_manager.cc:1905] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:37.647490 24770 ts_tablet_manager.cc:1918] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:37.647593 24770 log.cc:1196] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/ca8410df229846ce82a500ebdf239a75
I20260924 13:42:37.647854 24770 ts_tablet_manager.cc:1939] T ca8410df229846ce82a500ebdf239a75 P 2defa8306a5d4f1cb1e1b1d1de255b3e: Deleting consensus metadata
I20260924 13:42:37.648304 24267 catalog_manager.cc:4841] TS 2defa8306a5d4f1cb1e1b1d1de255b3e (127.19.53.194:46483): tablet ca8410df229846ce82a500ebdf239a75 (table test-table [id=64ae8fc48ebb40319814a63ec5979d8d]) successfully deleted
I20260924 13:42:37.650041 24768 tablet_replica.cc:330] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3: stopping tablet replica
I20260924 13:42:37.650146 24768 raft_consensus.cc:2229] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:42:37.650255 24768 raft_consensus.cc:2258] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:37.650621 24768 ts_tablet_manager.cc:1905] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:37.652065 24768 ts_tablet_manager.cc:1918] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:37.652133 24768 log.cc:1196] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/ca8410df229846ce82a500ebdf239a75
I20260924 13:42:37.652375 24768 ts_tablet_manager.cc:1939] T ca8410df229846ce82a500ebdf239a75 P 3e7c5ff888dd4618ac4808ea7e19a2b3: Deleting consensus metadata
I20260924 13:42:37.652750 24267 catalog_manager.cc:4841] TS 3e7c5ff888dd4618ac4808ea7e19a2b3 (127.19.53.195:34007): tablet ca8410df229846ce82a500ebdf239a75 (table test-table [id=64ae8fc48ebb40319814a63ec5979d8d]) successfully deleted
W20260924 13:42:37.653481 24265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:35217 (error 108)
I20260924 13:42:37.654063 24740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.653874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:35217 (local address 127.0.0.1:37064)
0924 13:42:37.653977 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:37.654036 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 13:42:37.654181 24265 catalog_manager.cc:4584] TS 8b4e795f390d475e9735eeb5c1d0fd5f (127.19.53.193:35217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca8410df229846ce82a500ebdf239a75: Network error: Client connection negotiation failed: client connection to 127.19.53.193:35217: connect: Connection refused (error 111)
I20260924 13:42:37.657989 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 24472
I20260924 13:42:37.662962 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 24603
I20260924 13:42:37.667622 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 24250
2026-09-24T13:42:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (635 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 13:42:37.687057 19671 mini_oidc.cc:152] Starting JWKS server
I20260924 13:42:37.688169 19671 webserver.cc:460] Webserver started at https://127.0.0.1:44505/ using document root <none> and password file <none>
I20260924 13:42:37.688292 19671 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 13:42:37.688476 19671 webserver.cc:460] Webserver started at http://127.0.0.1:35121/ using document root <none> and password file <none>
2026-09-24T13:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:37Z Disabled control of system clock
I20260924 13:42:37.696121 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:44865
--webserver_interface=127.19.53.254
--webserver_port=0
--builtin_ntp_servers=127.19.53.212:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:44865
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44505/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/testchainca.cert with env {}
W20260924 13:42:37.776157 24777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:37.776362 24777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:37.776384 24777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:37.777473 24777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:37.777510 24777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:37.777522 24777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:37.777535 24777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:37.777546 24777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 13:42:37.777559 24777 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 13:42:37.777570 24777 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44505/jwks.json
I20260924 13:42:37.779181 24777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:44865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:44865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44505/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:37.779378 24777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:37.779548 24777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:37.784300 24785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:37.784310 24783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:37.784332 24777 server_base.cc:1034] running on GCE node
W20260924 13:42:37.784315 24782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:37.784843 24777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:37.785121 24777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:37.786259 24777 hybrid_clock.cc:648] HybridClock initialized: now 1790257357786239 us; error 36 us; skew 500 ppm
I20260924 13:42:37.787942 24777 webserver.cc:460] Webserver started at http://127.19.53.254:38573/ using document root <none> and password file <none>
I20260924 13:42:37.788178 24777 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:37.788221 24777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:37.788307 24777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:37.789296 24777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "34606e809cf042f7bd10796495ac29f0"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "5095d2f94904fda1dd2fd17681f11b77"
server_key_iv: "365acd3023b4bae1932714037b4019a2"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.789580 24777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "34606e809cf042f7bd10796495ac29f0"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "5095d2f94904fda1dd2fd17681f11b77"
server_key_iv: "365acd3023b4bae1932714037b4019a2"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.790907 24777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:37.791568 24791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:37.791752 24777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:37.791852 24777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "34606e809cf042f7bd10796495ac29f0"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "5095d2f94904fda1dd2fd17681f11b77"
server_key_iv: "365acd3023b4bae1932714037b4019a2"
server_key_version: "encryptionkey@0"
I20260924 13:42:37.791919 24777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:37.803272 24777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:37.803563 24777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:37.803709 24777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:37.807392 24777 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:44865
I20260924 13:42:37.807422 24843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:44865 every 8 connection(s)
I20260924 13:42:37.807765 24777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:37.808723 24844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:37.810118 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 24777
I20260924 13:42:37.810222 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:37.810428 19671 external_mini_cluster.cc:1428] Setting key 7abff8d3632ed78bf705fb5cabdb315d
I20260924 13:42:37.811511 24844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0: Bootstrap starting.
I20260924 13:42:37.812237 24844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:37.812557 24844 log.cc:824] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:37.813359 24844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0: No bootstrap required, opened a new log
I20260924 13:42:37.815510 24844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34606e809cf042f7bd10796495ac29f0" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44865 } }
I20260924 13:42:37.815680 24844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:37.815727 24844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34606e809cf042f7bd10796495ac29f0, State: Initialized, Role: FOLLOWER
I20260924 13:42:37.815888 24844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34606e809cf042f7bd10796495ac29f0" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44865 } }
I20260924 13:42:37.815984 24844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:37.816023 24844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:37.816069 24844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:37.816637 24844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34606e809cf042f7bd10796495ac29f0" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44865 } }
I20260924 13:42:37.816781 24844 leader_election.cc:302] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34606e809cf042f7bd10796495ac29f0; no voters:
I20260924 13:42:37.817034 24844 leader_election.cc:288] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:37.817083 24849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:37.817231 24849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [term 1 LEADER]: Becoming Leader. State: Replica: 34606e809cf042f7bd10796495ac29f0, State: Running, Role: LEADER
I20260924 13:42:37.817323 24849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34606e809cf042f7bd10796495ac29f0" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44865 } }
I20260924 13:42:37.817341 24844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:37.817826 24850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34606e809cf042f7bd10796495ac29f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34606e809cf042f7bd10796495ac29f0" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44865 } } }
I20260924 13:42:37.817899 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.810934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.19.53.254:44865)
0924 13:42:37.811170 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:37.811179 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:37.811206 (+ 27us) server_negotiation.cc:407] Connection header received
0924 13:42:37.811736 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:37.811744 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:37.811829 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:37.811992 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:37.812219 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:37.813505 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:37.814343 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:37.814969 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:37.815149 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:37.816594 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:37.816597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:37.816598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:37.816820 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:37.816886 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:37.816903 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:37.817009 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:37.817145 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 13:42:37.817256 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":60,"threads_started":1}
I20260924 13:42:37.817901 24850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:37.818392 24851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34606e809cf042f7bd10796495ac29f0. Latest consensus state: current_term: 1 leader_uuid: "34606e809cf042f7bd10796495ac29f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34606e809cf042f7bd10796495ac29f0" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44865 } } }
I20260924 13:42:37.818739 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:42:37.818771 24858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:37.819276 24858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:37.819305 24851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:37.821048 24858 catalog_manager.cc:1347] Generated new cluster ID: ee8d8b5eda97450aa111ef1be6b27ac5
I20260924 13:42:37.821089 24858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:37.832406 24858 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:37.832891 24858 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:37.839273 24858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34606e809cf042f7bd10796495ac29f0: Generated new TSK 0
I20260924 13:42:37.839461 24858 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 13:42:37.863533 24874 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 13:42:37.863639 24874 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260924 13:42:37.864087 24873 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:37.849093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:44865 (local address 127.0.0.1:50740)
0924 13:42:37.849262 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:37.849279 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:37.849362 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:37.849547 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:37.849548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:37.849550 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 13:42:37.849636 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:37.849638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:37.850352 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:37.850354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:37.850950 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:37.850952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:37.851348 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:37.851349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:37.851412 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:37.851442 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 13:42:37.863771 (+ 12329us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 13:42:37.863853 (+ 82us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.53.254:44865: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260924 13:42:37.864424 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:37.849160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.19.53.254:44865)
0924 13:42:37.849225 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:37.849227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:37.849340 (+ 113us) server_negotiation.cc:407] Connection header received
0924 13:42:37.849447 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:37.849448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:37.849460 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:37.849539 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0924 13:42:37.849706 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:37.850272 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:37.851047 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:37.851250 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:37.851345 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:37.851567 (+ 222us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 13:42:37.863631 (+ 12064us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 13:42:37.864335 (+ 704us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50740: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 13:42:37.864497 24847 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50740: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.19.53.254:44865: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 13:42:37.864733 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 24777
2026-09-24T13:42:37Z chronyd exiting
I20260924 13:42:37.900933 19671 test_util.cc:175] -----------------------------------------------
I20260924 13:42:37.901015 19671 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257335418480-19671-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 13:42:37.901785 19671 mini_oidc.cc:152] Starting JWKS server
I20260924 13:42:37.902544 19671 webserver.cc:460] Webserver started at https://127.0.0.1:35531/ using document root <none> and password file <none>
I20260924 13:42:37.902585 19671 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 13:42:37.902746 19671 webserver.cc:460] Webserver started at http://127.0.0.1:35511/ using document root <none> and password file <none>
2026-09-24T13:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:37Z Disabled control of system clock
I20260924 13:42:37.911665 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:35867
--webserver_interface=127.19.53.254
--webserver_port=0
--builtin_ntp_servers=127.19.53.212:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:35867
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35531/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/testchainca.cert with env {}
W20260924 13:42:37.993064 24880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:37.993323 24880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:37.993362 24880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:37.994547 24880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:37.994616 24880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:37.994638 24880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:37.994658 24880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:37.994688 24880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 13:42:37.994714 24880 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 13:42:37.994740 24880 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35531/jwks.json
I20260924 13:42:37.996263 24880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:35867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:35867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35531/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:37.996538 24880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:37.996752 24880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:38.001513 24885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:38.001525 24886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:38.001555 24880 server_base.cc:1034] running on GCE node
W20260924 13:42:38.001518 24888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:38.001875 24880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:38.002190 24880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:38.003357 24880 hybrid_clock.cc:648] HybridClock initialized: now 1790257358003343 us; error 41 us; skew 500 ppm
I20260924 13:42:38.005069 24880 webserver.cc:460] Webserver started at http://127.19.53.254:45789/ using document root <none> and password file <none>
I20260924 13:42:38.005311 24880 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:38.005357 24880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:38.005503 24880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:38.006673 24880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "3a1c411aa5564a1594c6139e8cac91b1"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "9129f4593a3df080ebb27a2119244e9c"
server_key_iv: "db6073df55e2f332c6edfb69b31fe061"
server_key_version: "encryptionkey@0"
I20260924 13:42:38.007059 24880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "3a1c411aa5564a1594c6139e8cac91b1"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "9129f4593a3df080ebb27a2119244e9c"
server_key_iv: "db6073df55e2f332c6edfb69b31fe061"
server_key_version: "encryptionkey@0"
I20260924 13:42:38.008316 24880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:38.009032 24894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:38.009193 24880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:38.009297 24880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "3a1c411aa5564a1594c6139e8cac91b1"
format_stamp: "Formatted at 2026-09-24 13:42:37 on dist-test-slave-tfm9"
server_key: "9129f4593a3df080ebb27a2119244e9c"
server_key_iv: "db6073df55e2f332c6edfb69b31fe061"
server_key_version: "encryptionkey@0"
I20260924 13:42:38.009371 24880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:38.033190 24880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:38.033532 24880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:38.033695 24880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:38.037531 24880 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:35867
I20260924 13:42:38.037572 24946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:35867 every 8 connection(s)
I20260924 13:42:38.037961 24880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:38.038663 24947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:38.041332 24947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1: Bootstrap starting.
I20260924 13:42:38.041847 24947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:38.042160 24947 log.cc:824] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:38.042827 24947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1: No bootstrap required, opened a new log
I20260924 13:42:38.044324 24947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1c411aa5564a1594c6139e8cac91b1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35867 } }
I20260924 13:42:38.044461 24947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:38.044482 24947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a1c411aa5564a1594c6139e8cac91b1, State: Initialized, Role: FOLLOWER
I20260924 13:42:38.044624 24947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1c411aa5564a1594c6139e8cac91b1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35867 } }
I20260924 13:42:38.044703 24947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:38.044726 24947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:38.044754 24947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:38.045230 24947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1c411aa5564a1594c6139e8cac91b1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35867 } }
I20260924 13:42:38.045315 24947 leader_election.cc:302] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a1c411aa5564a1594c6139e8cac91b1; no voters:
I20260924 13:42:38.045437 24947 leader_election.cc:288] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:38.045586 24950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:38.045737 24950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [term 1 LEADER]: Becoming Leader. State: Replica: 3a1c411aa5564a1594c6139e8cac91b1, State: Running, Role: LEADER
I20260924 13:42:38.045755 24947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:38.045840 24950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1c411aa5564a1594c6139e8cac91b1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35867 } }
I20260924 13:42:38.046239 24952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a1c411aa5564a1594c6139e8cac91b1. Latest consensus state: current_term: 1 leader_uuid: "3a1c411aa5564a1594c6139e8cac91b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1c411aa5564a1594c6139e8cac91b1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35867 } } }
I20260924 13:42:38.046324 24952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:38.046478 24951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a1c411aa5564a1594c6139e8cac91b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1c411aa5564a1594c6139e8cac91b1" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35867 } } }
I20260924 13:42:38.046545 24951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:38.046823 24959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:38.047343 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 24880
I20260924 13:42:38.047448 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:38.047652 19671 external_mini_cluster.cc:1428] Setting key bb03de731017daaac198500b330e64b6
I20260924 13:42:38.047991 24959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:38.049439 24959 catalog_manager.cc:1347] Generated new cluster ID: c79e0ef46ad34bf6847015e9cb241680
I20260924 13:42:38.049494 24959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:38.055013 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:38.048128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.19.53.254:35867)
0924 13:42:38.048341 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:38.048351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:38.050117 (+ 1766us) server_negotiation.cc:407] Connection header received
0924 13:42:38.050256 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:38.050261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:38.050312 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:38.050435 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:38.050613 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.051467 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.052237 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.052644 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.052773 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:38.054201 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:38.054205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:38.054205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:38.054361 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:38.054393 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:38.054404 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:38.054475 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:38.054559 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 13:42:38.054639 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
I20260924 13:42:38.055676 24959 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:38.056088 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:42:38.057142 24959 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:38.061246 24959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a1c411aa5564a1594c6139e8cac91b1: Generated new TSK 0
I20260924 13:42:38.061439 24959 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 13:42:38.091296 24977 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 13:42:38.091387 24977 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 13:42:38.091671 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:38.066495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.19.53.254:35867)
0924 13:42:38.066562 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:38.066564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:38.069560 (+ 2996us) server_negotiation.cc:407] Connection header received
0924 13:42:38.069648 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:38.069649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:38.069668 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:38.069742 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0924 13:42:38.069881 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.070375 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.071154 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.071364 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.071450 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:38.071654 (+ 204us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 13:42:38.091446 (+ 19792us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 13:42:38.091615 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53584: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 13:42:38.091747 24968 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53584: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 13:42:38.091820 24976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:38.066444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:35867 (local address 127.0.0.1:53584)
0924 13:42:38.069488 (+ 3044us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:38.069497 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:38.069576 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:38.069741 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:38.069742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:38.069744 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 13:42:38.069805 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:38.069813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.070469 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.070470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:38.071070 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:38.071071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.071452 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.071453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:38.071517 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:38.071527 (+ 10us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 13:42:38.091603 (+ 20076us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 13:42:38.091717 (+ 114us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.53.254:35867: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":3024,"thread_start_us":89,"threads_started":1}
I20260924 13:42:38.092223 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 24880
2026-09-24T13:42:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 13:42:38.129768 19671 mini_oidc.cc:152] Starting JWKS server
I20260924 13:42:38.130532 19671 webserver.cc:460] Webserver started at https://127.0.0.1:39721/ using document root <none> and password file <none>
I20260924 13:42:38.130604 19671 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 13:42:38.130764 19671 webserver.cc:460] Webserver started at http://127.0.0.1:34113/ using document root <none> and password file <none>
2026-09-24T13:42:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:38Z Disabled control of system clock
I20260924 13:42:38.138196 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:43465
--webserver_interface=127.19.53.254
--webserver_port=0
--builtin_ntp_servers=127.19.53.212:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:43465
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39721/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 13:42:38.217746 24983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:38.217954 24983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:38.217983 24983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:38.219126 24983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:38.219166 24983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:38.219179 24983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:38.219216 24983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:38.219244 24983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 13:42:38.219266 24983 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 13:42:38.219292 24983 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39721/jwks.json
I20260924 13:42:38.220666 24983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:43465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:43465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39721/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:38.220911 24983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:38.221196 24983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:38.226074 24988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:38.226094 24989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:38.226163 24983 server_base.cc:1034] running on GCE node
W20260924 13:42:38.226060 24991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:38.226655 24983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:38.226940 24983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:38.228080 24983 hybrid_clock.cc:648] HybridClock initialized: now 1790257358228062 us; error 38 us; skew 500 ppm
I20260924 13:42:38.230195 24983 webserver.cc:460] Webserver started at http://127.19.53.254:34333/ using document root <none> and password file <none>
I20260924 13:42:38.230441 24983 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:38.230520 24983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:38.230655 24983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:38.231674 24983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "28ecd321a18248cfb56f3e62ff8c83c6"
format_stamp: "Formatted at 2026-09-24 13:42:38 on dist-test-slave-tfm9"
server_key: "b6080d319735a706f1e9644dc78ac9ac"
server_key_iv: "dc68e2f3e1c326490c15a7b71ac4ee73"
server_key_version: "encryptionkey@0"
I20260924 13:42:38.232029 24983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "28ecd321a18248cfb56f3e62ff8c83c6"
format_stamp: "Formatted at 2026-09-24 13:42:38 on dist-test-slave-tfm9"
server_key: "b6080d319735a706f1e9644dc78ac9ac"
server_key_iv: "dc68e2f3e1c326490c15a7b71ac4ee73"
server_key_version: "encryptionkey@0"
I20260924 13:42:38.233263 24983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:38.233982 24997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:38.234162 24983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:38.234269 24983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "28ecd321a18248cfb56f3e62ff8c83c6"
format_stamp: "Formatted at 2026-09-24 13:42:38 on dist-test-slave-tfm9"
server_key: "b6080d319735a706f1e9644dc78ac9ac"
server_key_iv: "dc68e2f3e1c326490c15a7b71ac4ee73"
server_key_version: "encryptionkey@0"
I20260924 13:42:38.234352 24983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:38.243579 24983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:38.243865 24983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:38.244014 24983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:38.247673 24983 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:43465
I20260924 13:42:38.247707 25049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:43465 every 8 connection(s)
I20260924 13:42:38.248023 24983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:38.248641 25050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:38.251329 25050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6: Bootstrap starting.
I20260924 13:42:38.251823 25050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:38.252134 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 24983
I20260924 13:42:38.252154 25050 log.cc:824] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:38.252246 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:38.252432 19671 external_mini_cluster.cc:1428] Setting key 9c22271bbd1f8d2cdbc34e67eda0e386
I20260924 13:42:38.252889 25050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6: No bootstrap required, opened a new log
I20260924 13:42:38.254575 25050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ecd321a18248cfb56f3e62ff8c83c6" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43465 } }
I20260924 13:42:38.254695 25050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:38.254719 25050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ecd321a18248cfb56f3e62ff8c83c6, State: Initialized, Role: FOLLOWER
I20260924 13:42:38.254865 25050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ecd321a18248cfb56f3e62ff8c83c6" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43465 } }
I20260924 13:42:38.254949 25050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:38.255010 25050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:38.255091 25050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:38.255613 25050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ecd321a18248cfb56f3e62ff8c83c6" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43465 } }
I20260924 13:42:38.255694 25050 leader_election.cc:302] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28ecd321a18248cfb56f3e62ff8c83c6; no voters:
I20260924 13:42:38.255860 25050 leader_election.cc:288] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:38.255885 25055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:38.255993 25055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [term 1 LEADER]: Becoming Leader. State: Replica: 28ecd321a18248cfb56f3e62ff8c83c6, State: Running, Role: LEADER
I20260924 13:42:38.256075 25055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ecd321a18248cfb56f3e62ff8c83c6" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43465 } }
I20260924 13:42:38.256114 25050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:38.256389 25056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28ecd321a18248cfb56f3e62ff8c83c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ecd321a18248cfb56f3e62ff8c83c6" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43465 } } }
I20260924 13:42:38.256395 25057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28ecd321a18248cfb56f3e62ff8c83c6. Latest consensus state: current_term: 1 leader_uuid: "28ecd321a18248cfb56f3e62ff8c83c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ecd321a18248cfb56f3e62ff8c83c6" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43465 } } }
I20260924 13:42:38.256546 25056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:38.256564 25057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6 [sys.catalog]: This master's current role is: LEADER
W20260924 13:42:38.258116 25071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:38.258204 25071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:38.258255 25061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:38.258401 25061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:38.260811 25061 catalog_manager.cc:1347] Generated new cluster ID: f3fbcd0460d3470db53f6b9001dea940
I20260924 13:42:38.260866 25061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:38.261263 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:38.252905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.19.53.254:43465)
0924 13:42:38.253100 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:38.253107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:38.253129 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:42:38.254562 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:38.254569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:38.254638 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:38.254788 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:38.255020 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.256725 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.258064 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.258708 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.258856 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:38.260564 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:38.260568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:38.260568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:38.260756 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:38.260797 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:38.260811 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:38.260871 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:38.260985 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 13:42:38.261064 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":62,"threads_started":1}
I20260924 13:42:38.262172 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:42:38.278450 25061 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:38.279150 25061 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:38.286000 25061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28ecd321a18248cfb56f3e62ff8c83c6: Generated new TSK 0
I20260924 13:42:38.286170 25061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:42:38.298623 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:38.294804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.19.53.254:43465)
0924 13:42:38.294863 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:38.294865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:38.294957 (+ 92us) server_negotiation.cc:407] Connection header received
0924 13:42:38.295025 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:38.295026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:38.295036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:38.295094 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0924 13:42:38.295239 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.295732 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.296893 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.297185 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.297225 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:38.297511 (+ 286us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 13:42:38.298451 (+ 940us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39721/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 13:42:38.298561 (+ 110us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43910: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39721/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 13:42:38.298681 25054 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43910: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39721/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 13:42:38.301067 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:38.294687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:43465 (local address 127.0.0.1:43910)
0924 13:42:38.294885 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:38.294895 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:38.294958 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:38.295115 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:38.295116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:38.295118 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 13:42:38.295175 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:38.295177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.295805 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.295806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:38.296792 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:38.296795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.297293 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.297294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:38.297349 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:38.297353 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 13:42:38.298785 (+ 1432us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39721/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 13:42:38.300973 (+ 2188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.53.254:43465: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39721/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":2003,"thread_start_us":125,"threads_started":1}
I20260924 13:42:38.301445 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 24983
E20260924 13:42:38.309885 25080 webserver.cc:533] Webserver: sslize failed:
E20260924 13:42:38.309991 25080 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T13:42:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (209 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T13:42:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:38Z Disabled control of system clock
I20260924 13:42:38.345867 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:46603
--webserver_interface=127.19.53.254
--webserver_port=0
--builtin_ntp_servers=127.19.53.212:35593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:46603
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/testchainca.cert with env {}
W20260924 13:42:38.426640 25084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:38.426836 25084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:38.426858 25084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:38.427948 25084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:38.427987 25084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:38.428001 25084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:38.428016 25084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:38.428030 25084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 13:42:38.428045 25084 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 13:42:38.428061 25084 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 13:42:38.429577 25084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:35593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:46603
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:46603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:38.429762 25084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:38.429947 25084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:38.434849 25089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:38.434859 25090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:38.434854 25092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:38.435146 25084 server_base.cc:1034] running on GCE node
I20260924 13:42:38.435372 25084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:38.435628 25084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:38.436770 25084 hybrid_clock.cc:648] HybridClock initialized: now 1790257358436757 us; error 27 us; skew 500 ppm
I20260924 13:42:38.438671 25084 webserver.cc:460] Webserver started at http://127.19.53.254:43553/ using document root <none> and password file <none>
I20260924 13:42:38.438918 25084 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:38.438966 25084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:38.439110 25084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:38.440187 25084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "540b958621394f3d944d02468e973d8c"
format_stamp: "Formatted at 2026-09-24 13:42:38 on dist-test-slave-tfm9"
server_key: "ae1d878fa6b38d6feb8f0e090bd8197c"
server_key_iv: "d298b14580f0b4f5a71f6847cd226c46"
server_key_version: "encryptionkey@0"
I20260924 13:42:38.440544 25084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "540b958621394f3d944d02468e973d8c"
format_stamp: "Formatted at 2026-09-24 13:42:38 on dist-test-slave-tfm9"
server_key: "ae1d878fa6b38d6feb8f0e090bd8197c"
server_key_iv: "d298b14580f0b4f5a71f6847cd226c46"
server_key_version: "encryptionkey@0"
I20260924 13:42:38.441808 25084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:38.442554 25098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:38.442740 25084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:38.442823 25084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "540b958621394f3d944d02468e973d8c"
format_stamp: "Formatted at 2026-09-24 13:42:38 on dist-test-slave-tfm9"
server_key: "ae1d878fa6b38d6feb8f0e090bd8197c"
server_key_iv: "d298b14580f0b4f5a71f6847cd226c46"
server_key_version: "encryptionkey@0"
I20260924 13:42:38.442898 25084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:38.457119 25084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:38.457398 25084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:38.457623 25084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:38.461390 25084 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:46603
I20260924 13:42:38.461468 25150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:46603 every 8 connection(s)
I20260924 13:42:38.461779 25084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:38.462690 25151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:38.464874 25151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c: Bootstrap starting.
I20260924 13:42:38.465418 25151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:38.465703 25151 log.cc:824] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:38.466423 25151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c: No bootstrap required, opened a new log
I20260924 13:42:38.468006 25151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540b958621394f3d944d02468e973d8c" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 46603 } }
I20260924 13:42:38.468149 25151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:38.468212 25151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540b958621394f3d944d02468e973d8c, State: Initialized, Role: FOLLOWER
I20260924 13:42:38.468380 25151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540b958621394f3d944d02468e973d8c" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 46603 } }
I20260924 13:42:38.468443 25151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:38.468463 25151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:38.468537 25151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:38.469048 25151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540b958621394f3d944d02468e973d8c" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 46603 } }
I20260924 13:42:38.469156 25151 leader_election.cc:302] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 540b958621394f3d944d02468e973d8c; no voters:
I20260924 13:42:38.469329 25151 leader_election.cc:288] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:38.469372 25154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:38.469575 25151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:38.469585 25154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [term 1 LEADER]: Becoming Leader. State: Replica: 540b958621394f3d944d02468e973d8c, State: Running, Role: LEADER
I20260924 13:42:38.469949 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 25084
I20260924 13:42:38.470049 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:38.470239 19671 external_mini_cluster.cc:1428] Setting key 8437ada58c99a745c1a5242321f23356
I20260924 13:42:38.469712 25154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540b958621394f3d944d02468e973d8c" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 46603 } }
I20260924 13:42:38.470929 25155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "540b958621394f3d944d02468e973d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540b958621394f3d944d02468e973d8c" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 46603 } } }
I20260924 13:42:38.471114 25156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 540b958621394f3d944d02468e973d8c. Latest consensus state: current_term: 1 leader_uuid: "540b958621394f3d944d02468e973d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540b958621394f3d944d02468e973d8c" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 46603 } } }
W20260924 13:42:38.471446 25169 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 13:42:38.471462 25155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [sys.catalog]: This master's current role is: LEADER
W20260924 13:42:38.471522 25169 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:38.471475 25156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:38.471729 25172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:38.471902 25172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:38.473423 25172 catalog_manager.cc:1347] Generated new cluster ID: 542a6610d05940718406dc4c48825516
I20260924 13:42:38.473488 25172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:38.476948 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:38.470900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.19.53.254:46603)
0924 13:42:38.471245 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:38.471254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:38.471418 (+ 164us) server_negotiation.cc:407] Connection header received
0924 13:42:38.471541 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:38.471549 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:38.471629 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:38.471818 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:38.472052 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.473350 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.474291 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.474723 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.474865 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:38.476266 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:38.476269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:38.476270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:38.476488 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:38.476520 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:38.476531 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:38.476604 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:38.476677 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 13:42:38.476757 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":55,"threads_started":1}
I20260924 13:42:38.477826 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:42:38.490175 25172 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:38.490641 25172 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:38.497080 25172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 540b958621394f3d944d02468e973d8c: Generated new TSK 0
I20260924 13:42:38.497256 25172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:42:38.539657 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:38.506031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.19.53.254:46603)
0924 13:42:38.506098 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:38.506100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:38.506255 (+ 155us) server_negotiation.cc:407] Connection header received
0924 13:42:38.506335 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:38.506336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:38.506347 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:38.506406 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0924 13:42:38.506567 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.507063 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.508220 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.508391 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.508435 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:38.508633 (+ 198us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 13:42:38.539446 (+ 30813us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 13:42:38.539601 (+ 155us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46486: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 13:42:38.539742 25170 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46486: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 13:42:38.539870 25180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:38.506007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:46603 (local address 127.0.0.1:46486)
0924 13:42:38.506177 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:38.506190 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:38.506261 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:38.506424 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:38.506425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:38.506427 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 13:42:38.506493 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:38.506496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.507140 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.507142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:38.508113 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:38.508116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:38.508465 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:38.508467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:38.508527 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:38.508534 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 13:42:38.539600 (+ 31066us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 13:42:38.539759 (+ 159us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.53.254:46603: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":100,"threads_started":1}
I20260924 13:42:38.540292 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 25084
2026-09-24T13:42:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (217 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:38 dist-test-slave-tfm9 krb5kdc[25192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:38 dist-test-slave-tfm9 krb5kdc[25192](info): set up 2 sockets
Sep 24 13:42:38 dist-test-slave-tfm9 krb5kdc[25192](info): commencing operation
krb5kdc: starting...
W20260924 13:42:39.740078 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:39 dist-test-slave-tfm9 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:42:39.857296 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:44911
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:44911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:39.943164 25205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:39.943394 25205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:39.943418 25205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:39.944458 25205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:39.944494 25205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:39.944509 25205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:39.944523 25205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:39.944536 25205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:39.945852 25205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:44911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:44911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:39.946120 25205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:39.946336 25205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:39.951213 25211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:39.951253 25213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:39.951431 25205 server_base.cc:1034] running on GCE node
W20260924 13:42:39.951244 25210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:39.951771 25205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:39.952070 25205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:39.953230 25205 hybrid_clock.cc:648] HybridClock initialized: now 1790257359953217 us; error 26 us; skew 500 ppm
Sep 24 13:42:39 dist-test-slave-tfm9 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257359, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:39.956274 25205 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:42:39.956715 25205 webserver.cc:460] Webserver started at http://127.19.53.254:33087/ using document root <none> and password file <none>
I20260924 13:42:39.956933 25205 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:39.956976 25205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:39.957119 25205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:39.958240 25205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "95d93e5b8f3640e4ae29577bead424f5"
format_stamp: "Formatted at 2026-09-24 13:42:39 on dist-test-slave-tfm9"
server_key: "20cdc31ff38073108fa7373653777b69"
server_key_iv: "767d1983029a274f4a3c8bc6b6a2e7a4"
server_key_version: "encryptionkey@0"
I20260924 13:42:39.958595 25205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "95d93e5b8f3640e4ae29577bead424f5"
format_stamp: "Formatted at 2026-09-24 13:42:39 on dist-test-slave-tfm9"
server_key: "20cdc31ff38073108fa7373653777b69"
server_key_iv: "767d1983029a274f4a3c8bc6b6a2e7a4"
server_key_version: "encryptionkey@0"
I20260924 13:42:39.959885 25205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:39.960616 25220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:39.960791 25205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:39.960877 25205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "95d93e5b8f3640e4ae29577bead424f5"
format_stamp: "Formatted at 2026-09-24 13:42:39 on dist-test-slave-tfm9"
server_key: "20cdc31ff38073108fa7373653777b69"
server_key_iv: "767d1983029a274f4a3c8bc6b6a2e7a4"
server_key_version: "encryptionkey@0"
I20260924 13:42:39.960948 25205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:39.978655 25205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:39.982180 25205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:39.982286 25205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:39.985881 25205 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:44911
I20260924 13:42:39.985977 25272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:44911 every 8 connection(s)
I20260924 13:42:39.986311 25205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:39.987124 25273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:39.989370 25273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5: Bootstrap starting.
I20260924 13:42:39.990015 25273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:39.990301 25273 log.cc:824] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:39.990947 25273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5: No bootstrap required, opened a new log
I20260924 13:42:39.991660 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 25205
I20260924 13:42:39.991752 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:39.991955 19671 external_mini_cluster.cc:1428] Setting key 0ae7e935d9aa593aa58d1d1c795d5143
I20260924 13:42:39.993077 25273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d93e5b8f3640e4ae29577bead424f5" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44911 } }
I20260924 13:42:39.993252 25273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:39.993299 25273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d93e5b8f3640e4ae29577bead424f5, State: Initialized, Role: FOLLOWER
I20260924 13:42:39.993424 25273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d93e5b8f3640e4ae29577bead424f5" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44911 } }
I20260924 13:42:39.993506 25273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:39.993551 25273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:39.993623 25273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:39.994532 25273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d93e5b8f3640e4ae29577bead424f5" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44911 } }
I20260924 13:42:39.994650 25273 leader_election.cc:302] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95d93e5b8f3640e4ae29577bead424f5; no voters:
I20260924 13:42:39.994834 25273 leader_election.cc:288] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:39.994867 25278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:39.994975 25278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [term 1 LEADER]: Becoming Leader. State: Replica: 95d93e5b8f3640e4ae29577bead424f5, State: Running, Role: LEADER
I20260924 13:42:39.995097 25273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:39.995067 25278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d93e5b8f3640e4ae29577bead424f5" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44911 } }
I20260924 13:42:39.995411 25278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d93e5b8f3640e4ae29577bead424f5. Latest consensus state: current_term: 1 leader_uuid: "95d93e5b8f3640e4ae29577bead424f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d93e5b8f3640e4ae29577bead424f5" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44911 } } }
I20260924 13:42:39.995486 25278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:39.995700 25284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:39.996205 25284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:39.996510 25279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95d93e5b8f3640e4ae29577bead424f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d93e5b8f3640e4ae29577bead424f5" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 44911 } } }
I20260924 13:42:39.996579 25279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:39.998100 25284 catalog_manager.cc:1347] Generated new cluster ID: 365c54a070704bdf8680db811a22b270
I20260924 13:42:39.998152 25284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:42:39 dist-test-slave-tfm9 krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:40.004930 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:39.992459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53002 (local address 127.19.53.254:44911)
0924 13:42:39.992646 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:39.992654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:39.992690 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:42:39.993050 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:39.993069 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:39.993140 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:39.993282 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:39.994147 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:39.995319 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:39.996313 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:39.996837 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:39.996974 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:40.000133 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:40.000137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:40.000137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:40.000149 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:40.003443 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.003817 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.003820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.003821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.003923 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.004140 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.004141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.004141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.004371 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:40.004553 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:40.004617 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 13:42:40.004754 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260924 13:42:40.012274 25284 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:40.012933 25284 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:40.029778 25284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95d93e5b8f3640e4ae29577bead424f5: Generated new TSK 0
I20260924 13:42:40.030019 25284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:42:40.054769 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:44911
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:40.141652 25299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:40.141872 25299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:40.141909 25299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:40.143016 25299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:40.143077 25299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:40.143121 25299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:40.144542 25299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:44911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:40.144794 25299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:40.145010 25299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:42:40.150130 25299 server_base.cc:1034] running on GCE node
W20260924 13:42:40.150110 25305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:40.150101 25304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:40.150117 25307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:40.150506 25299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:40.150753 25299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:40.151935 25299 hybrid_clock.cc:648] HybridClock initialized: now 1790257360151913 us; error 36 us; skew 500 ppm
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:40.155206 25299 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:42:40.155653 25299 webserver.cc:460] Webserver started at http://127.19.53.193:44985/ using document root <none> and password file <none>
I20260924 13:42:40.155823 25299 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:40.155860 25299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:40.155946 25299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:40.156881 25299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "a9bb93dfd61d46908594ddd005674dc5"
format_stamp: "Formatted at 2026-09-24 13:42:40 on dist-test-slave-tfm9"
server_key: "4645b258307f16e1ff20cd040ce476d7"
server_key_iv: "4922b8ca7b586be48f0cbcccc5b8d936"
server_key_version: "encryptionkey@0"
I20260924 13:42:40.157161 25299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "a9bb93dfd61d46908594ddd005674dc5"
format_stamp: "Formatted at 2026-09-24 13:42:40 on dist-test-slave-tfm9"
server_key: "4645b258307f16e1ff20cd040ce476d7"
server_key_iv: "4922b8ca7b586be48f0cbcccc5b8d936"
server_key_version: "encryptionkey@0"
I20260924 13:42:40.158561 25299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:40.159263 25314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:40.159443 25299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:40.159544 25299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "a9bb93dfd61d46908594ddd005674dc5"
format_stamp: "Formatted at 2026-09-24 13:42:40 on dist-test-slave-tfm9"
server_key: "4645b258307f16e1ff20cd040ce476d7"
server_key_iv: "4922b8ca7b586be48f0cbcccc5b8d936"
server_key_version: "encryptionkey@0"
I20260924 13:42:40.159598 25299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:40.196506 25299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:40.199802 25299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:40.199923 25299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:40.200103 25299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:40.200407 25299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:40.200438 25299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:40.200481 25299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:40.200508 25299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:40.205834 25299 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:45301
I20260924 13:42:40.205927 25427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:45301 every 8 connection(s)
I20260924 13:42:40.206261 25299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:42:40.210386 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 25299
I20260924 13:42:40.210512 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:42:40.210702 19671 external_mini_cluster.cc:1428] Setting key 6c6f98721a553ccbd50ae72e26ce5cfd
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:40.218652 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:40.206983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:35517 (local address 127.19.53.254:44911)
0924 13:42:40.207045 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:40.207047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:40.207284 (+ 237us) server_negotiation.cc:407] Connection header received
0924 13:42:40.207833 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:40.207835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:40.207849 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:40.207887 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:40.208815 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.209370 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.210479 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.210749 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.210832 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:40.214102 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:40.214104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:40.214104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:40.214112 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:40.217173 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.217569 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.217571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.217571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.217638 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.217871 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.217872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.217873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.218117 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:40.218194 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:40.218444 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 13:42:40.218541 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:42:40.218681 25430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:40.206979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:44911 (local address 127.19.53.193:35517)
0924 13:42:40.207170 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:40.207203 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:40.207716 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:40.207982 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:40.207985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:40.208239 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:40.208705 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:40.208709 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.209488 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.209489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:40.210379 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:40.210381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.210845 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.210847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:40.211030 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:40.212325 (+ 1295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:40.212335 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:40.214005 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:40.217268 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:40.217274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:40.217276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:40.217487 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:40.217711 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:40.217711 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:40.217712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:40.217807 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:40.218188 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:40.218193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:40.218308 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 13:42:40.218395 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 13:42:40.218523 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":96,"threads_started":1}
I20260924 13:42:40.219195 25428 heartbeater.cc:347] Connected to a master server at 127.19.53.254:44911
I20260924 13:42:40.219288 25428 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:40.219476 25428 heartbeater.cc:511] Master 127.19.53.254:44911 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
I20260924 13:42:40.220043 25237 ts_manager.cc:194] Registered new tserver with Master: a9bb93dfd61d46908594ddd005674dc5 (127.19.53.193:45301)
I20260924 13:42:40.220641 25237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:35517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:42:40.236635 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:44911
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:40.317276 25433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:40.317504 25433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:40.317543 25433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:40.318840 25433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:40.318909 25433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:40.318950 25433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:40.320411 25433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:44911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:40.320681 25433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:40.320901 25433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:40.325824 25439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:40.325901 25438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:40.325812 25441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:40.325984 25433 server_base.cc:1034] running on GCE node
I20260924 13:42:40.326246 25433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:40.326500 25433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:40.327669 25433 hybrid_clock.cc:648] HybridClock initialized: now 1790257360327653 us; error 48 us; skew 500 ppm
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:40.331064 25433 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:42:40.331489 25433 webserver.cc:460] Webserver started at http://127.19.53.194:41963/ using document root <none> and password file <none>
I20260924 13:42:40.331710 25433 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:40.331779 25433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:40.331925 25433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:40.332953 25433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "b8e4184db1544d40a79289cd47f23f23"
format_stamp: "Formatted at 2026-09-24 13:42:40 on dist-test-slave-tfm9"
server_key: "2f53595a1532e343236511cc0136f544"
server_key_iv: "f17511cd5a5d793ef1fc12718a1da2af"
server_key_version: "encryptionkey@0"
I20260924 13:42:40.333314 25433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "b8e4184db1544d40a79289cd47f23f23"
format_stamp: "Formatted at 2026-09-24 13:42:40 on dist-test-slave-tfm9"
server_key: "2f53595a1532e343236511cc0136f544"
server_key_iv: "f17511cd5a5d793ef1fc12718a1da2af"
server_key_version: "encryptionkey@0"
I20260924 13:42:40.334713 25433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:40.335534 25448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:40.335750 25433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:42:40.335855 25433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "b8e4184db1544d40a79289cd47f23f23"
format_stamp: "Formatted at 2026-09-24 13:42:40 on dist-test-slave-tfm9"
server_key: "2f53595a1532e343236511cc0136f544"
server_key_iv: "f17511cd5a5d793ef1fc12718a1da2af"
server_key_version: "encryptionkey@0"
I20260924 13:42:40.335927 25433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:40.349551 25433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:40.353752 25433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:40.353952 25433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:40.354204 25433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:40.354501 25433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:40.354557 25433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:40.354593 25433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:40.354619 25433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:40.360358 25433 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:35457
I20260924 13:42:40.360397 25561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:35457 every 8 connection(s)
I20260924 13:42:40.360755 25433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:40.370817 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 25433
I20260924 13:42:40.370900 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:42:40.371126 19671 external_mini_cluster.cc:1428] Setting key 057973703f18c969094f3be62b1cdf6e
I20260924 13:42:40.371853 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:40.361539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:56749 (local address 127.19.53.254:44911)
0924 13:42:40.361626 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:40.361628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:40.361793 (+ 165us) server_negotiation.cc:407] Connection header received
0924 13:42:40.362390 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:40.362392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:40.362403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:40.362465 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:40.363428 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.364000 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.365033 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.365260 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.365347 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:40.368199 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:40.368200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:40.368201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:40.368208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:40.370391 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.370816 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.370817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.370818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.370904 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.371187 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.371187 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.371188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.371319 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:40.371398 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:40.371667 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 13:42:40.371754 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 13:42:40.372047 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:40.361497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:44911 (local address 127.19.53.194:56749)
0924 13:42:40.361671 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:40.361712 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:40.362289 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:40.362536 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:40.362540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:40.362781 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:40.363296 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:40.363301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.364094 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.364095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:40.364923 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:40.364925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.365358 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.365359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:40.365481 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:40.366748 (+ 1267us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:40.366756 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:40.368089 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:40.370490 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:40.370497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:40.370499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:40.370728 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:40.371000 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:40.371001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:40.371001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:40.371104 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:40.371388 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:40.371394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:40.371534 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 13:42:40.371645 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 13:42:40.371795 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260924 13:42:40.372536 25562 heartbeater.cc:347] Connected to a master server at 127.19.53.254:44911
I20260924 13:42:40.372629 25562 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:40.372810 25562 heartbeater.cc:511] Master 127.19.53.254:44911 requested a full tablet report, sending...
I20260924 13:42:40.373181 25237 ts_manager.cc:194] Registered new tserver with Master: b8e4184db1544d40a79289cd47f23f23 (127.19.53.194:35457)
I20260924 13:42:40.373634 25237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:56749
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:42:40.395018 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:44911
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:40.473873 25567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:40.474103 25567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:40.474125 25567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:40.475220 25567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:40.475265 25567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:40.475315 25567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:40.476727 25567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:44911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:40.476955 25567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:40.477192 25567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:40.482506 25573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:40.482543 25575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:40.482548 25572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:40.482954 25567 server_base.cc:1034] running on GCE node
I20260924 13:42:40.483179 25567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:40.483444 25567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:40.484606 25567 hybrid_clock.cc:648] HybridClock initialized: now 1790257360484595 us; error 30 us; skew 500 ppm
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:40.487532 25567 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:42:40.488026 25567 webserver.cc:460] Webserver started at http://127.19.53.195:38293/ using document root <none> and password file <none>
I20260924 13:42:40.488281 25567 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:40.488327 25567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:40.488487 25567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:40.489513 25567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "3e7b618077624c9da761757eb911e1bf"
format_stamp: "Formatted at 2026-09-24 13:42:40 on dist-test-slave-tfm9"
server_key: "04aa6a0bbf3bc26aa0c21b1e6bc7e17f"
server_key_iv: "53ae9fb4e6f247e6076ff974177daa9b"
server_key_version: "encryptionkey@0"
I20260924 13:42:40.489884 25567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "3e7b618077624c9da761757eb911e1bf"
format_stamp: "Formatted at 2026-09-24 13:42:40 on dist-test-slave-tfm9"
server_key: "04aa6a0bbf3bc26aa0c21b1e6bc7e17f"
server_key_iv: "53ae9fb4e6f247e6076ff974177daa9b"
server_key_version: "encryptionkey@0"
I20260924 13:42:40.491256 25567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:40.492055 25582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:40.492221 25567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:40.492307 25567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "3e7b618077624c9da761757eb911e1bf"
format_stamp: "Formatted at 2026-09-24 13:42:40 on dist-test-slave-tfm9"
server_key: "04aa6a0bbf3bc26aa0c21b1e6bc7e17f"
server_key_iv: "53ae9fb4e6f247e6076ff974177daa9b"
server_key_version: "encryptionkey@0"
I20260924 13:42:40.492385 25567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:40.517033 25567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:40.520336 25567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:40.520503 25567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:40.520743 25567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:40.521071 25567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:40.521128 25567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:40.521171 25567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:40.521204 25567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:40.527187 25567 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:44607
I20260924 13:42:40.527230 25695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:44607 every 8 connection(s)
I20260924 13:42:40.527550 25567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:42:40.529472 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 25567
I20260924 13:42:40.529582 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:42:40.529745 19671 external_mini_cluster.cc:1428] Setting key 2e8040219511e8408ae8313441edcb55
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:40.537906 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:40.528270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:56595 (local address 127.19.53.254:44911)
0924 13:42:40.528324 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:40.528327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:40.528506 (+ 179us) server_negotiation.cc:407] Connection header received
0924 13:42:40.528960 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:40.528961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:40.528971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:40.529027 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:40.530176 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.530915 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.531960 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.532136 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.532186 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:40.534696 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:40.534697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:40.534698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:40.534703 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:40.536760 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.537132 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.537133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.537133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.537181 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.537387 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.537387 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.537388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.537482 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:40.537550 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:40.537781 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 13:42:40.537842 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:40.538156 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:40.528205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:44911 (local address 127.19.53.195:56595)
0924 13:42:40.528382 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:40.528418 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:40.528866 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:40.529081 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:40.529084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:40.529345 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:40.530053 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:40.530060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.531000 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.531001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:40.531889 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:40.531892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.532204 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.532205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:40.532301 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:40.533221 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:40.533230 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:40.534605 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:40.536831 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:40.536836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:40.536838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:40.537065 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:40.537238 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:40.537239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:40.537239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:40.537333 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:40.537555 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:40.537558 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:40.537671 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 13:42:40.537750 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 13:42:40.537865 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260924 13:42:40.538668 25696 heartbeater.cc:347] Connected to a master server at 127.19.53.254:44911
I20260924 13:42:40.538785 25696 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:40.538955 25696 heartbeater.cc:511] Master 127.19.53.254:44911 requested a full tablet report, sending...
I20260924 13:42:40.539394 25237 ts_manager.cc:194] Registered new tserver with Master: 3e7b618077624c9da761757eb911e1bf (127.19.53.195:44607)
I20260924 13:42:40.539848 25237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:56595
I20260924 13:42:40.541320 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/test-admin.keytab.
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:40.558084 19671 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 13:42:40.559708 25703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:40.558957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.19.53.193:45301)
0924 13:42:40.559146 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:40.559151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:40.559161 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:42:40.559189 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:40.559191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:40.559205 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:40.559271 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:40.559626 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57534: BlockingRecv error: recv got EOF from 127.0.0.1:57534 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":99,"threads_started":1}
W20260924 13:42:40.559906 19671 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/krb5cc' of type FILE
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:40.561009 19671 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:40.562067 19671 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.193@KRBTEST.COM
I20260924 13:42:40.571087 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:40.562600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.19.53.193:45301)
0924 13:42:40.562656 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:40.562659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:40.562723 (+ 64us) server_negotiation.cc:407] Connection header received
0924 13:42:40.562794 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:40.562795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:40.562806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:40.562878 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:40.563325 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.564054 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.564805 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.565132 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.565213 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:40.567769 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:40.567776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:40.567777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:40.567784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:40.570108 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.570423 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.570426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.570426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.570479 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.570664 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.570665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.570665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.570793 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:40.570892 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:40.570961 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 13:42:40.571023 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 13:42:40.572889 25703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:40.572287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.19.53.193:45301)
0924 13:42:40.572341 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:40.572343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:40.572392 (+ 49us) server_negotiation.cc:407] Connection header received
0924 13:42:40.572462 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:40.572463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:40.572472 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:40.572558 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:40.572842 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57558: BlockingRecv error: recv got EOF from 127.0.0.1:57558 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 13:42:40.572964 19671 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/krb5cc' of type FILE
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:40.573879 19671 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:40.574761 19671 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.193@KRBTEST.COM
I20260924 13:42:40.582721 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:40.575204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.19.53.193:45301)
0924 13:42:40.575259 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:40.575261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:40.575310 (+ 49us) server_negotiation.cc:407] Connection header received
0924 13:42:40.575381 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:40.575382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:40.575391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:40.575459 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:40.575820 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.576289 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.576964 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.577166 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.577241 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:40.579638 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:40.579639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:40.579639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:40.579646 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:40.581645 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.582010 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.582011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.582012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.582069 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.582262 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.582263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.582264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.582356 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:40.582466 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:40.582554 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 13:42:40.582628 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 13:42:40.584040 25703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:40.583542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.19.53.193:45301)
0924 13:42:40.583587 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:40.583589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:40.583673 (+ 84us) server_negotiation.cc:407] Connection header received
0924 13:42:40.583710 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:40.583711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:40.583718 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:40.583763 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:40.583976 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57584: BlockingRecv error: recv got EOF from 127.0.0.1:57584 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 13:42:40.584136 19671 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257335418480-19671-0/krb5kdc/krb5cc' of type FILE
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:40.584968 19671 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:40.585768 19671 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.193@KRBTEST.COM
I20260924 13:42:40.593384 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:40.586285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.19.53.193:45301)
0924 13:42:40.586332 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:40.586334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:40.586377 (+ 43us) server_negotiation.cc:407] Connection header received
0924 13:42:40.586444 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:40.586445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:40.586453 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:40.586518 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:40.586873 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.587305 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.587994 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:40.588190 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:40.588266 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:40.590641 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:40.590642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:40.590642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:40.590648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:40.592524 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.592807 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.592808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.592808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.592851 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:40.592989 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:40.592989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:40.592989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:40.593059 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:40.593118 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:40.593199 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 13:42:40.593247 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 13:42:40.594044 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 25299
I20260924 13:42:40.598989 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 25433
I20260924 13:42:40.604038 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 25567
I20260924 13:42:40.608271 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 25205
2026-09-24T13:42:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1900 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25716](info): set up 2 sockets
Sep 24 13:42:40 dist-test-slave-tfm9 krb5kdc[25716](info): commencing operation
krb5kdc: starting...
W20260924 13:42:41.627833 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:41 dist-test-slave-tfm9 krb5kdc[25716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.254.keytab.
Entry for principal oryx/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.254.keytab.
I20260924 13:42:41.741732 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:34111
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.254.keytab
--principal=oryx/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:34111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:41.821597 25729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:41.821799 25729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:41.821821 25729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:41.822949 25729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:41.822990 25729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:41.823004 25729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:41.823015 25729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:41.823027 25729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:41.824369 25729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:34111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.254.keytab
--principal=oryx/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:34111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:41.824546 25729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:41.824702 25729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:41.829631 25734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:41.829617 25729 server_base.cc:1034] running on GCE node
W20260924 13:42:41.829617 25737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:41.829666 25735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:41.830019 25729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:41.830358 25729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:41.831516 25729 hybrid_clock.cc:648] HybridClock initialized: now 1790257361831509 us; error 35 us; skew 500 ppm
Sep 24 13:42:41 dist-test-slave-tfm9 krb5kdc[25716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257361, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:41.842664 25729 init.cc:377] Logged in from keytab as oryx/127.19.53.254@KRBTEST.COM (short username oryx)
I20260924 13:42:41.843185 25729 webserver.cc:460] Webserver started at http://127.19.53.254:42115/ using document root <none> and password file <none>
I20260924 13:42:41.843439 25729 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:41.843510 25729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:41.843672 25729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:41.844765 25729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "8ee85e1707af4b85bd01c90c229805a2"
format_stamp: "Formatted at 2026-09-24 13:42:41 on dist-test-slave-tfm9"
server_key: "de9e5d2133c70393654f47d198fd2603"
server_key_iv: "10eab43198d1439afc729bee3b857ce8"
server_key_version: "encryptionkey@0"
I20260924 13:42:41.845155 25729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "8ee85e1707af4b85bd01c90c229805a2"
format_stamp: "Formatted at 2026-09-24 13:42:41 on dist-test-slave-tfm9"
server_key: "de9e5d2133c70393654f47d198fd2603"
server_key_iv: "10eab43198d1439afc729bee3b857ce8"
server_key_version: "encryptionkey@0"
I20260924 13:42:41.846637 25729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:41.847371 25744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:41.847570 25729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:41.847666 25729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "8ee85e1707af4b85bd01c90c229805a2"
format_stamp: "Formatted at 2026-09-24 13:42:41 on dist-test-slave-tfm9"
server_key: "de9e5d2133c70393654f47d198fd2603"
server_key_iv: "10eab43198d1439afc729bee3b857ce8"
server_key_version: "encryptionkey@0"
I20260924 13:42:41.847723 25729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:41.858853 25729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:41.860991 25729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:41.861111 25729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:41.864967 25729 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:34111
I20260924 13:42:41.865021 25796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:34111 every 8 connection(s)
I20260924 13:42:41.865388 25729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:41.866056 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 25729
I20260924 13:42:41.866187 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:41.866401 19671 external_mini_cluster.cc:1428] Setting key f4b4770b19ed29b94f656dfbb2d70c29
I20260924 13:42:41.866631 25797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:41.869418 25797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2: Bootstrap starting.
I20260924 13:42:41.870100 25797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:41.870396 25797 log.cc:824] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:41.871068 25797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2: No bootstrap required, opened a new log
Sep 24 13:42:41 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:41.872871 25797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } }
I20260924 13:42:41.873027 25797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:41.873071 25797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee85e1707af4b85bd01c90c229805a2, State: Initialized, Role: FOLLOWER
I20260924 13:42:41.873255 25797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } }
I20260924 13:42:41.873375 25797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:41.873416 25797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:41.873464 25797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:41.874092 25797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } }
I20260924 13:42:41.874212 25797 leader_election.cc:302] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ee85e1707af4b85bd01c90c229805a2; no voters:
I20260924 13:42:41.874387 25797 leader_election.cc:288] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:41.874419 25802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:41.874531 25802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee85e1707af4b85bd01c90c229805a2, State: Running, Role: LEADER
I20260924 13:42:41.874619 25802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } }
I20260924 13:42:41.874643 25797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:41.876200 25803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } }
I20260924 13:42:41.876499 25803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:41.876338 25804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee85e1707af4b85bd01c90c229805a2. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } }
I20260924 13:42:41.876739 25804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: This master's current role is: LEADER
W20260924 13:42:41.876961 25816 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:41.877027 25816 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:41.877091 25818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:41.877259 25818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:41.877417 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:41.866891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.19.53.254:34111)
0924 13:42:41.867079 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:41.867088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:41.867122 (+ 34us) server_negotiation.cc:407] Connection header received
0924 13:42:41.867737 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:41.867743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:41.867791 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:41.867932 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:41.868592 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:41.869475 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:41.870317 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:41.870780 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:41.870911 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:41.873192 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:41.873196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:41.873196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:41.873208 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:41.875321 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:41.875730 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:41.875732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:41.875733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:41.875810 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:41.876066 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:41.876067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:41.876068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:41.876486 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:41.876637 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:41.877080 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 13:42:41.877213 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260924 13:42:41.878842 25818 catalog_manager.cc:1347] Generated new cluster ID: 2e5a03513b4546cd9964bbd188ba433d
I20260924 13:42:41.878903 25818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:41.888419 25818 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:41.889134 25818 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:41.891810 25818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2: Generated new TSK 0
I20260924 13:42:41.891983 25818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab.
Entry for principal oryx/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab.
I20260924 13:42:41.925559 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34111
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab
--principal=oryx/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:42.006672 25823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:42.006857 25823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:42.006879 25823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:42.007928 25823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:42.007967 25823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:42.007992 25823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:42.009339 25823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab
--principal=oryx/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:42.009514 25823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:42.009689 25823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:42.014748 25828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:42.014818 25829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:42.014871 25831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:42.014765 25823 server_base.cc:1034] running on GCE node
I20260924 13:42:42.015102 25823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:42.015378 25823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:42.016554 25823 hybrid_clock.cc:648] HybridClock initialized: now 1790257362016566 us; error 49 us; skew 500 ppm
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:42.019068 25823 init.cc:377] Logged in from keytab as oryx/127.19.53.193@KRBTEST.COM (short username oryx)
I20260924 13:42:42.019487 25823 webserver.cc:460] Webserver started at http://127.19.53.193:42223/ using document root <none> and password file <none>
I20260924 13:42:42.019732 25823 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:42.019814 25823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:42.019977 25823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:42.021018 25823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "aded32df8cd54b0d9b2861430781094e"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "15836c72c31cc8ac15c9303772dfbfd3"
server_key_iv: "01dc8be526e05f52f1b0d2c427a26cc7"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.021373 25823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "aded32df8cd54b0d9b2861430781094e"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "15836c72c31cc8ac15c9303772dfbfd3"
server_key_iv: "01dc8be526e05f52f1b0d2c427a26cc7"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.022764 25823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:42.023464 25838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.023633 25823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:42.023725 25823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "aded32df8cd54b0d9b2861430781094e"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "15836c72c31cc8ac15c9303772dfbfd3"
server_key_iv: "01dc8be526e05f52f1b0d2c427a26cc7"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.023808 25823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:42.044449 25823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:42.046671 25823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:42.046842 25823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:42.047096 25823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:42.047405 25823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:42.047492 25823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.047554 25823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:42.047582 25823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.053464 25823 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:39725
I20260924 13:42:42.053560 25951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:39725 every 8 connection(s)
I20260924 13:42:42.053851 25823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:42.060611 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 25823
I20260924 13:42:42.060722 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:42:42.060910 19671 external_mini_cluster.cc:1428] Setting key 3fa94658e936e2863fe31a1d58f595f9
I20260924 13:42:42.063402 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.054630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:54099 (local address 127.19.53.254:34111)
0924 13:42:42.054710 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.054712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.054842 (+ 130us) server_negotiation.cc:407] Connection header received
0924 13:42:42.055333 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.055334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.055344 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.055397 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.056289 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.056842 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.057830 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.058037 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.058089 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.060178 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.060179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.060179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.060185 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.061875 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.062339 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.062341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.062341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.062405 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.062659 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.062660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.062661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.062803 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.062880 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.063170 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.063267 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 13:42:42.063413 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.054570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.193:54099)
0924 13:42:42.054709 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.054758 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.055248 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.055451 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.055454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.056018 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.056211 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.056216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.056925 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.056925 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.057743 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.057745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.058130 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.058131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.058241 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.058674 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.058681 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.060095 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.062011 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.062018 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.062020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.062245 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.062472 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.062473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.062473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.062587 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.062886 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.062894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.063033 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.063125 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.063243 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260924 13:42:42.063863 25952 heartbeater.cc:347] Connected to a master server at 127.19.53.254:34111
I20260924 13:42:42.063977 25952 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:42.064198 25952 heartbeater.cc:511] Master 127.19.53.254:34111 requested a full tablet report, sending...
I20260924 13:42:42.064764 25761 ts_manager.cc:194] Registered new tserver with Master: aded32df8cd54b0d9b2861430781094e (127.19.53.193:39725)
I20260924 13:42:42.065376 25761 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:54099
WARNING: no policy specified for oryx/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab.
Entry for principal oryx/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab.
I20260924 13:42:42.086694 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34111
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab
--principal=oryx/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:42.165709 25957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:42.165902 25957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:42.165940 25957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:42.167114 25957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:42.167156 25957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:42.167210 25957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:42.168627 25957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab
--principal=oryx/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:42.168865 25957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:42.169126 25957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:42.174134 25963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:42.174144 25965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:42.174144 25962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:42.174186 25957 server_base.cc:1034] running on GCE node
I20260924 13:42:42.174477 25957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:42.174726 25957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:42.175870 25957 hybrid_clock.cc:648] HybridClock initialized: now 1790257362175835 us; error 50 us; skew 500 ppm
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:42.178427 25957 init.cc:377] Logged in from keytab as oryx/127.19.53.194@KRBTEST.COM (short username oryx)
I20260924 13:42:42.178889 25957 webserver.cc:460] Webserver started at http://127.19.53.194:46235/ using document root <none> and password file <none>
I20260924 13:42:42.179113 25957 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:42.179158 25957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:42.179319 25957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:42.180343 25957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "8de5b4cde81d40c3bcee0982b73f5573"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "2de85aa9d4cfa1ad805c401eebc381de"
server_key_iv: "b1e0cd12ea65fb4769908d94e68dd3a3"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.180681 25957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "8de5b4cde81d40c3bcee0982b73f5573"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "2de85aa9d4cfa1ad805c401eebc381de"
server_key_iv: "b1e0cd12ea65fb4769908d94e68dd3a3"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.182057 25957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:42:42.182797 25972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.182969 25957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.183061 25957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "8de5b4cde81d40c3bcee0982b73f5573"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "2de85aa9d4cfa1ad805c401eebc381de"
server_key_iv: "b1e0cd12ea65fb4769908d94e68dd3a3"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.183144 25957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:42.230049 25957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:42.232173 25957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:42.232334 25957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:42.232575 25957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:42.232915 25957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:42.232970 25957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.233014 25957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:42.233042 25957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.239106 25957 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:37865
I20260924 13:42:42.239147 26085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:37865 every 8 connection(s)
I20260924 13:42:42.239463 25957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:42:42.241681 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 25957
I20260924 13:42:42.241786 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:42:42.242031 19671 external_mini_cluster.cc:1428] Setting key 07c27083fee58b87aa766a34c1e9abf4
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:42.250214 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.240175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:53933 (local address 127.19.53.254:34111)
0924 13:42:42.240256 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.240259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.240422 (+ 163us) server_negotiation.cc:407] Connection header received
0924 13:42:42.240905 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.240907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.240918 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.240967 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.242269 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.242907 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.244450 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.244776 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.244853 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.247329 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.247331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.247331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.247338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.248755 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.249189 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.249190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.249191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.249251 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.249502 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.249503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.249503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.249616 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.249695 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.250041 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.250121 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for oryx/127.19.53.195@KRBTEST.COM; defaulting to no policy
I20260924 13:42:42.250411 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.240075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.194:53933)
0924 13:42:42.240222 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.240269 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.240822 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.241032 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.241036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.241795 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.242134 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.242141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.242996 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.242997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.244337 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.244340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.244874 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.244875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.244989 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.245572 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.245580 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.247228 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.248874 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.248881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.248883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.249111 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.249319 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.249320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.249320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.249431 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.249686 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.249692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.249862 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.250019 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.250176 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260924 13:42:42.250861 26086 heartbeater.cc:347] Connected to a master server at 127.19.53.254:34111
I20260924 13:42:42.250985 26086 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:42.251178 26086 heartbeater.cc:511] Master 127.19.53.254:34111 requested a full tablet report, sending...
I20260924 13:42:42.251606 25761 ts_manager.cc:194] Registered new tserver with Master: 8de5b4cde81d40c3bcee0982b73f5573 (127.19.53.194:37865)
I20260924 13:42:42.251998 25761 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:53933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab.
Entry for principal oryx/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab.
I20260924 13:42:42.269143 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34111
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab
--principal=oryx/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:42.352630 26091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:42.352828 26091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:42.352852 26091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:42.354120 26091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:42.354166 26091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:42.354223 26091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:42.355661 26091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab
--principal=oryx/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:42.355849 26091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:42.356045 26091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:42.361302 26099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:42.361346 26096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:42.361567 26091 server_base.cc:1034] running on GCE node
W20260924 13:42:42.361302 26097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:42.361812 26091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:42.362093 26091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:42.363261 26091 hybrid_clock.cc:648] HybridClock initialized: now 1790257362363248 us; error 51 us; skew 500 ppm
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:42.365731 26091 init.cc:377] Logged in from keytab as oryx/127.19.53.195@KRBTEST.COM (short username oryx)
I20260924 13:42:42.366245 26091 webserver.cc:460] Webserver started at http://127.19.53.195:44979/ using document root <none> and password file <none>
I20260924 13:42:42.366466 26091 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:42.366513 26091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:42.366607 26091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:42.367611 26091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "3e51301c17dd4526937621aeba3fed60"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "150f32fc75057340b9929303f960e3f6"
server_key_iv: "6793bef67712ceb0e871ffcba069efa0"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.367933 26091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "3e51301c17dd4526937621aeba3fed60"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "150f32fc75057340b9929303f960e3f6"
server_key_iv: "6793bef67712ceb0e871ffcba069efa0"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.369230 26091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:42.370015 26106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.370165 26091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:42.370245 26091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "3e51301c17dd4526937621aeba3fed60"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "150f32fc75057340b9929303f960e3f6"
server_key_iv: "6793bef67712ceb0e871ffcba069efa0"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.370298 26091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:42.407137 26091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:42.408999 26091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:42.409108 26091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:42.409294 26091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:42.409626 26091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:42.409659 26091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.409705 26091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:42.409731 26091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.415613 26091 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:35269
I20260924 13:42:42.415771 26219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:35269 every 8 connection(s)
I20260924 13:42:42.416021 26091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:42.424136 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 26091
I20260924 13:42:42.424226 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:42:42.424417 19671 external_mini_cluster.cc:1428] Setting key 3f2518d65f2f596a93b8b929d34ac9dc
I20260924 13:42:42.424747 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.416747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:37553 (local address 127.19.53.254:34111)
0924 13:42:42.416826 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.416829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.416982 (+ 153us) server_negotiation.cc:407] Connection header received
0924 13:42:42.417433 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.417434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.417444 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.417497 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.418478 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.418971 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.419943 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.420148 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.420200 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.422271 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.422272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.422272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.422278 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.423441 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.423791 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.423792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.423792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.423839 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.424069 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.424070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.424070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.424219 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.424290 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.424578 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.424656 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 13:42:42.424998 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.416706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.195:37553)
0924 13:42:42.416855 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.416890 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.417352 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.417551 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.417554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.418203 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.418402 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.418407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.419066 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.419066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.419869 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.419871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.420217 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.420218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.420316 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.420827 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.420835 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.422188 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.423516 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.423521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.423523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.423729 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.423895 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.423896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.423896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.423985 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.424293 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.424298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.424418 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.424541 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.424693 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260924 13:42:42.425416 26220 heartbeater.cc:347] Connected to a master server at 127.19.53.254:34111
I20260924 13:42:42.425529 26220 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:42.425698 26220 heartbeater.cc:511] Master 127.19.53.254:34111 requested a full tablet report, sending...
I20260924 13:42:42.426116 25761 ts_manager.cc:194] Registered new tserver with Master: 3e51301c17dd4526937621aeba3fed60 (127.19.53.195:35269)
I20260924 13:42:42.426486 25761 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:37553
I20260924 13:42:42.436096 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.53.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.53.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.53.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.253.keytab.
Entry for principal oryx/127.19.53.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.253.keytab.
I20260924 13:42:42.460383 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.253:41489
--webserver_interface=127.19.53.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.253.keytab
--principal=oryx/127.19.53.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:34111,127.19.53.253:41489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:42.540473 26225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:42.540671 26225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:42.540692 26225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:42.541836 26225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:42.541877 26225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:42.541891 26225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:42.541949 26225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:42.541975 26225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:42.543442 26225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:34111,127.19.53.253:41489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.253.keytab
--principal=oryx/127.19.53.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.253:41489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.53.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
W20260924 13:42:42.543537 26225 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.19.53.254:34111,127.19.53.253:41489'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 13:42:42.543702 26225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:42.543938 26225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:42.548724 26233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:42.548715 26231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:42.548761 26230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:42.549059 26225 server_base.cc:1034] running on GCE node
I20260924 13:42:42.549286 26225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:42.549620 26225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:42.550812 26225 hybrid_clock.cc:648] HybridClock initialized: now 1790257362550795 us; error 30 us; skew 500 ppm
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:42.553552 26225 init.cc:377] Logged in from keytab as oryx/127.19.53.253@KRBTEST.COM (short username oryx)
I20260924 13:42:42.554066 26225 webserver.cc:460] Webserver started at http://127.19.53.253:43729/ using document root <none> and password file <none>
I20260924 13:42:42.554309 26225 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:42.554378 26225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:42.554540 26225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:42.555572 26225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/data/instance:
uuid: "7e5fa5965fd84e7f964d462f6ae37efe"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "e8c9ea15473ec4fed2f7c5dad1bbdb2b"
server_key_iv: "9b35c7ffa715ba7d3b0f29306c527e13"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.555924 26225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/wal/instance:
uuid: "7e5fa5965fd84e7f964d462f6ae37efe"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "e8c9ea15473ec4fed2f7c5dad1bbdb2b"
server_key_iv: "9b35c7ffa715ba7d3b0f29306c527e13"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.557200 26225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:42.558003 26240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.558171 26225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:42.558270 26225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/wal
uuid: "7e5fa5965fd84e7f964d462f6ae37efe"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "e8c9ea15473ec4fed2f7c5dad1bbdb2b"
server_key_iv: "9b35c7ffa715ba7d3b0f29306c527e13"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.558351 26225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:42.567720 26225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:42.570176 26225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:42.570333 26225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:42.574209 26225 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.253:41489
I20260924 13:42:42.574242 26292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.253:41489 every 8 connection(s)
I20260924 13:42:42.574584 26225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/data/info.pb
I20260924 13:42:42.574860 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 26225
I20260924 13:42:42.575902 26293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:42.577874 26293 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.53.253@KRBTEST.COM
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.253@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:42.585383 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.578570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.19.53.254:34111)
0924 13:42:42.578632 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.578634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.578814 (+ 180us) server_negotiation.cc:407] Connection header received
0924 13:42:42.578870 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.578871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.578880 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.578923 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.579426 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.579892 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.580849 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.581054 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.581127 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.582918 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.582919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.582920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.582926 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.584134 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.584508 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.584509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.584510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.584565 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.584788 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.584789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.584790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.584936 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.585023 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.585204 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.585288 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:42:42.585680 26297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.578499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:45504)
0924 13:42:42.578694 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.578744 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.578810 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.578939 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.578942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.579273 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.579348 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.579350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.580007 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.580008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.580751 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.580754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.581133 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.581134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.581209 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.581624 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.581632 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.582831 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.584217 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.584223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.584225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.584416 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.584630 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.584631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.584632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.584721 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.585013 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.585018 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.585081 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.585185 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.585392 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
I20260924 13:42:42.585685 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.575527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.19.53.253:41489)
0924 13:42:42.575735 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.575744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.575775 (+ 31us) server_negotiation.cc:407] Connection header received
0924 13:42:42.576434 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.576442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.576503 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.576669 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.577237 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.578662 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.579694 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.580093 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.580209 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.582405 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.582411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.582411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.582417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.584122 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.584473 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.584475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.584476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.584556 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.584765 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.584765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.584766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.584957 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.585117 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.585187 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.585466 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260924 13:42:42.586369 26293 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.253@KRBTEST.COM for oryx/127.19.53.253@KRBTEST.COM
I20260924 13:42:42.591787 26297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.586588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:41489 (local address 127.0.0.1:48748)
0924 13:42:42.586632 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.586642 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.586679 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.586772 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.586773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.586873 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.586898 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.586899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.587536 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.587537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.588006 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.588008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.588307 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.588308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.588353 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.588636 (+ 283us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.588639 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.589657 (+ 1018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.591090 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.591090 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.591091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.591229 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.591375 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.591376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.591376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.591419 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.591578 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.591578 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.591607 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.591654 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.591717 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 13:42:42.591835 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.586610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.19.53.253:41489)
0924 13:42:42.586657 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.586658 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.586667 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:42.586725 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.586725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.586731 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.586771 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.586942 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.587464 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.588068 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.588255 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.588303 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.589725 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.589726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.589726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.589732 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.591028 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.591286 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.591287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.591287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.591325 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.591456 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.591457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.591457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.591532 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.591572 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.591669 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.591755 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":30}
I20260924 13:42:42.592995 26293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: Bootstrap starting.
I20260924 13:42:42.593768 26293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:42.594098 26293 log.cc:824] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:42.594769 26293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: No bootstrap required, opened a new log
I20260924 13:42:42.596112 26293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } }
I20260924 13:42:42.596242 26293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:42.596266 26293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5fa5965fd84e7f964d462f6ae37efe, State: Initialized, Role: FOLLOWER
I20260924 13:42:42.596421 26293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } }
I20260924 13:42:42.596665 26299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } } }
I20260924 13:42:42.596700 26293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:42.596764 26299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: This master's current role is: FOLLOWER
W20260924 13:42:42.598251 26310 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:42.598310 26310 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:42.599697 26225 master_runner.cc:428] Detected that this master 7e5fa5965fd84e7f964d462f6ae37efe is joining an existing cluster
I20260924 13:42:42.599731 26225 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.53.253:41489
I20260924 13:42:42.606825 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.600554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:45508)
0924 13:42:42.600895 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.600905 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.600965 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.601129 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.601130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.601383 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.601452 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.601454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.602143 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.602144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.602717 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.602718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.603080 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.603081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.603138 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.603598 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.603603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.603884 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.605450 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.605451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.605451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.605636 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.605878 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.605879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.605879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.605991 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.606438 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.606440 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.606481 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.606586 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.606701 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":82,"threads_started":1}
I20260924 13:42:42.606827 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.600601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.19.53.254:34111)
0924 13:42:42.600644 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.600646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.600945 (+ 299us) server_negotiation.cc:407] Connection header received
0924 13:42:42.601026 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.601027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.601035 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.601125 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.601531 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.602068 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.602799 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.602982 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.603056 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.604017 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.604019 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.604019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.604026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.605369 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.605732 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.605734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.605734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.605792 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.606246 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.606247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.606247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.606353 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.606435 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.606627 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.606716 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:42:42.608755 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.600801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.19.53.253:41489)
0924 13:42:42.600842 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.600844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.601117 (+ 273us) server_negotiation.cc:407] Connection header received
0924 13:42:42.601201 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.601201 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.601211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.601264 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.601717 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.602428 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.603444 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.603700 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.603735 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.605817 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.605818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.605818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.605824 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.607563 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.607925 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.607926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.607927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.607986 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.608240 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.608241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.608242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.608371 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.608422 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.608598 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.608669 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:42:42.609045 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.600742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:41489 (local address 127.0.0.1:48752)
0924 13:42:42.601073 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.601081 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.601134 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.601268 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.601268 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.601534 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.601617 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.601619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.602520 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.602521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.603360 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.603362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.603797 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.603797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.603853 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.604278 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.604283 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.605715 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.607647 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.607648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.607648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.607840 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.608065 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.608066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.608066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.608153 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.608436 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.608437 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.608487 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.608855 (+ 368us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.608932 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":59,"threads_started":1}
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257361, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.254@KRBTEST.COM for oryx/127.19.53.253@KRBTEST.COM
I20260924 13:42:42.616230 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.609253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.19.53.253:41489)
0924 13:42:42.609284 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.609286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.609338 (+ 52us) server_negotiation.cc:407] Connection header received
0924 13:42:42.609423 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.609424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.609436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.609502 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.609970 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.610386 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.611518 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.611726 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.611776 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.613687 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.613689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.613689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.613696 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.615053 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.615451 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.615452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.615453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.615511 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.615715 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.615716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.615716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.615816 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.615889 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.616073 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.616144 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:42:42.616276 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.609102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:41489 (local address 127.0.0.1:48764)
0924 13:42:42.609239 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.609283 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.609359 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.609510 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.609513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.609822 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.609876 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.609879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.610467 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.610468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.611425 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.611428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.611799 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.611800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.611882 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.612337 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.612341 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.613600 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.615155 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.615162 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.615164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.615362 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.615579 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.615580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.615580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.615651 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.615883 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.615891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.615941 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.616046 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.616160 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260924 13:42:42.616859 25761 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.53.253:41489: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe"
member_type: NON_VOTER
last_known_addr {
host: "127.19.53.253"
port: 41489
}
attrs {
promote: true
}
}
dest_uuid: "8ee85e1707af4b85bd01c90c229805a2"
cas_config_opid_index: -1
I20260924 13:42:42.617170 25761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true } }
I20260924 13:42:42.617614 26319 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7e5fa5965fd84e7f964d462f6ae37efe (127.19.53.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true } } }
I20260924 13:42:42.617937 26267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:42.618567 26319 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.53.253:41489
I20260924 13:42:42.618738 26319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true } } }
I20260924 13:42:42.618786 26225 master.cc:537] Master@127.19.53.253:41489 shutting down...
I20260924 13:42:42.618816 26319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:42.618851 26267 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 FOLLOWER]: Refusing update from remote peer 8ee85e1707af4b85bd01c90c229805a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 13:42:42.619273 26320 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 13:42:42.620138 25747 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 13:42:42.620829 25747 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 -> Peer 7e5fa5965fd84e7f964d462f6ae37efe (127.19.53.253:41489): Couldn't send request to peer 7e5fa5965fd84e7f964d462f6ae37efe. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 13:42:42.621590 26225 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:42.621680 26225 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:42.621729 26225 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: stopping tablet replica
I20260924 13:42:42.634087 26225 master.cc:552] Master@127.19.53.253:41489 shutdown complete.
I20260924 13:42:42.634130 26225 master_runner.cc:305] Clearing existing system tablet
I20260924 13:42:42.634274 26225 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:42.634310 26225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:42.634912 26225 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:42.635445 26324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.635574 26225 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:42.635635 26225 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.635670 26225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/wal
uuid: "7e5fa5965fd84e7f964d462f6ae37efe"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "e8c9ea15473ec4fed2f7c5dad1bbdb2b"
server_key_iv: "9b35c7ffa715ba7d3b0f29306c527e13"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.636256 26225 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:42.637657 26225 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 13:42:42.637713 26225 log.cc:1196] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 13:42:42.637995 26225 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: Deleting consensus metadata
I20260924 13:42:42.638129 26225 master_runner.cc:315] Copying system tablet from 127.19.53.254:34111
I20260924 13:42:42.638733 26225 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: tablet copy: Beginning tablet copy session from remote peer at address 127.19.53.254:34111
I20260924 13:42:42.644677 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.639097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.19.53.254:34111)
0924 13:42:42.639141 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.639144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.639261 (+ 117us) server_negotiation.cc:407] Connection header received
0924 13:42:42.639343 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.639344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.639356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.639413 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.639738 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.640233 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.640968 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.641177 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.641251 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.642268 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.642270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.642270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.642277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.643547 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.643881 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.643882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.643883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.643941 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.644170 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.644171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.644172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.644278 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.644353 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.644520 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.644589 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:42.644688 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.639037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:45520)
0924 13:42:42.639179 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.639188 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.639278 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.639419 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.639420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.639597 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.639653 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.639655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.640315 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.640316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.640883 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.640884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.641250 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.641250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.641331 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.641839 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.641843 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.642174 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.643631 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.643632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.643633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.643807 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.644021 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.644022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.644022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.644081 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.644345 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.644346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.644389 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.644478 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.644582 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260924 13:42:42.645203 25781 tablet_copy_service.cc:140] P 8ee85e1707af4b85bd01c90c229805a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7e5fa5965fd84e7f964d462f6ae37efe ({username='oryx', principal='oryx/127.19.53.253@KRBTEST.COM'} at 127.0.0.1:45520)
I20260924 13:42:42.645280 25781 tablet_copy_service.cc:161] P 8ee85e1707af4b85bd01c90c229805a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7e5fa5965fd84e7f964d462f6ae37efe at {username='oryx', principal='oryx/127.19.53.253@KRBTEST.COM'} at 127.0.0.1:45520: session id = 7e5fa5965fd84e7f964d462f6ae37efe-00000000000000000000000000000000
I20260924 13:42:42.645957 25781 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 13:42:42.646636 26225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 13:42:42.647521 26225 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: tablet copy: Starting download of 0 data blocks...
I20260924 13:42:42.647670 26225 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: tablet copy: Starting download of 1 WAL segments...
I20260924 13:42:42.649544 26225 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 13:42:42.650480 25781 tablet_copy_service.cc:342] P 8ee85e1707af4b85bd01c90c229805a2: Request end of tablet copy session 7e5fa5965fd84e7f964d462f6ae37efe-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.53.253@KRBTEST.COM'} at 127.0.0.1:45520
I20260924 13:42:42.650556 25781 tablet_copy_service.cc:433] P 8ee85e1707af4b85bd01c90c229805a2: ending tablet copy session 7e5fa5965fd84e7f964d462f6ae37efe-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7e5fa5965fd84e7f964d462f6ae37efe
I20260924 13:42:42.652820 26225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:42.654078 26331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:42.654076 26332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:42.654292 26334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:42.654488 26225 server_base.cc:1034] running on GCE node
I20260924 13:42:42.654578 26225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:42.654716 26225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:42.655846 26225 hybrid_clock.cc:648] HybridClock initialized: now 1790257362655845 us; error 31 us; skew 500 ppm
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:42.657111 26225 init.cc:377] Logged in from keytab as oryx/127.19.53.253@KRBTEST.COM (short username oryx)
I20260924 13:42:42.657373 26225 webserver.cc:460] Webserver started at http://127.19.53.253:46513/ using document root <none> and password file <none>
I20260924 13:42:42.657518 26225 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:42.657567 26225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:42.658210 26225 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:42.658896 26341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.659058 26225 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:42.659107 26225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/wal
uuid: "7e5fa5965fd84e7f964d462f6ae37efe"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "e8c9ea15473ec4fed2f7c5dad1bbdb2b"
server_key_iv: "9b35c7ffa715ba7d3b0f29306c527e13"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.659164 26225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:42.673668 26225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:42.675861 26225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:42.678884 26225 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.253:41489
I20260924 13:42:42.679070 26393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.253:41489 every 8 connection(s)
I20260924 13:42:42.679265 26225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-1/data/info.pb
I20260924 13:42:42.679984 26394 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:42:42.680395 26394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: Bootstrap starting.
I20260924 13:42:42.682612 26394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:42:42.682837 26394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: Bootstrap complete.
I20260924 13:42:42.683048 26394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true } }
I20260924 13:42:42.683116 26394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e5fa5965fd84e7f964d462f6ae37efe, State: Initialized, Role: LEARNER
I20260924 13:42:42.683142 26394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true } }
I20260924 13:42:42.683321 26394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:42.683328 26397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true } } }
I20260924 13:42:42.683462 26397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: This master's current role is: LEARNER
I20260924 13:42:42.684736 26408 catalog_manager.cc:1259] Loaded cluster ID: 2e5a03513b4546cd9964bbd188ba433d
I20260924 13:42:42.684780 26408 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: loading cluster ID for follower catalog manager: success
I20260924 13:42:42.685810 26408 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: acquiring CA information for follower catalog manager: success
I20260924 13:42:42.686098 26408 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:42:42.694636 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.687652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.19.53.253:41489)
0924 13:42:42.687744 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.687746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.687757 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:42:42.687788 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.687788 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.687800 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.687883 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.688406 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.689420 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.690227 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.690551 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.690636 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.691792 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.691802 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.691802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.691809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.693468 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.693803 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.693804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.693805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.693871 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.694118 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.694119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.694119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.694252 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.694358 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.694453 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.694532 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"thread_start_us":44,"threads_started":1}
WARNING: no policy specified for oryx/127.19.53.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.53.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.53.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.252.keytab.
Entry for principal oryx/127.19.53.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.252.keytab.
I20260924 13:42:42.720346 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.252:36097
--webserver_interface=127.19.53.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.252.keytab
--principal=oryx/127.19.53.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:34111,127.19.53.253:41489,127.19.53.252:36097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:42.802919 26412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:42.803112 26412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:42.803133 26412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:42.804154 26412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:42.804188 26412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:42.804201 26412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:42.804214 26412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:42.804225 26412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:42.805677 26412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:34111,127.19.53.253:41489,127.19.53.252:36097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.252.keytab
--principal=oryx/127.19.53.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.252:36097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.53.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:42.805871 26412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:42.806097 26412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:42.810781 26417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:42.810770 26420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:42.810854 26418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:42.811015 26412 server_base.cc:1034] running on GCE node
I20260924 13:42:42.811278 26412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:42.811532 26412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:42.812704 26412 hybrid_clock.cc:648] HybridClock initialized: now 1790257362812687 us; error 29 us; skew 500 ppm
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:42.815183 26412 init.cc:377] Logged in from keytab as oryx/127.19.53.252@KRBTEST.COM (short username oryx)
I20260924 13:42:42.815604 26412 webserver.cc:460] Webserver started at http://127.19.53.252:44573/ using document root <none> and password file <none>
I20260924 13:42:42.815827 26412 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:42.815892 26412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:42.816048 26412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:42.817075 26412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/data/instance:
uuid: "68acfc076b894170aa626af998518c5a"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "1137352e9cb96d81b7c3774e275c95a2"
server_key_iv: "3b2719703f61647b6a47738017c38258"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.817433 26412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/wal/instance:
uuid: "68acfc076b894170aa626af998518c5a"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "1137352e9cb96d81b7c3774e275c95a2"
server_key_iv: "3b2719703f61647b6a47738017c38258"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.818876 26412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:42.819644 26427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.819825 26412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:42.819934 26412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/wal
uuid: "68acfc076b894170aa626af998518c5a"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "1137352e9cb96d81b7c3774e275c95a2"
server_key_iv: "3b2719703f61647b6a47738017c38258"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.820024 26412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:42.837129 26412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:42.839071 26412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:42.839232 26412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:42.843184 26412 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.252:36097
I20260924 13:42:42.843219 26479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.252:36097 every 8 connection(s)
I20260924 13:42:42.843602 26412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/data/info.pb
I20260924 13:42:42.844628 26480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:42.845021 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 26412
I20260924 13:42:42.846926 26480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.53.252@KRBTEST.COM
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.252@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:42.855688 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.847664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.19.53.254:34111)
0924 13:42:42.847714 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.847716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.847758 (+ 42us) server_negotiation.cc:407] Connection header received
0924 13:42:42.847861 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.847862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.847874 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.847921 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.848368 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.848864 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.849821 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.850173 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.850244 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.852801 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.852802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.852803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.852810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.854366 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.854823 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.854824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.854825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.854878 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.855117 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.855117 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.855118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.855219 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.855271 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.855501 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.855580 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:42:42.855875 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.847504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:45536)
0924 13:42:42.847666 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.847706 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.847783 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.847947 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.847949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.848190 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.848280 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.848284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.848957 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.848958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.849735 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.849738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.850291 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.850292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.850365 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.850901 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.850911 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.852714 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.854491 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.854496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.854498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.854755 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.854950 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.854950 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.854951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.855051 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.855285 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.855291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.855348 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.855441 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.855684 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":106,"thread_start_us":86,"threads_started":1}
I20260924 13:42:42.855875 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.845648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.19.53.252:36097)
0924 13:42:42.845775 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.845784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.845806 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:42:42.846293 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.846300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.846349 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.846537 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.847114 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.848298 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.849088 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.849490 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.849564 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.852316 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.852324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.852325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.852331 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.854221 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.854620 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.854623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.854623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.854712 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.854968 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.854969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.854970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.855232 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.855377 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.855513 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.855638 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":57,"threads_started":1}
I20260924 13:42:42.856460 26480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.252@KRBTEST.COM for oryx/127.19.53.253@KRBTEST.COM
I20260924 13:42:42.862686 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.856697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:41489 (local address 127.0.0.1:48774)
0924 13:42:42.856730 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.856740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.856790 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.856994 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.856995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.857128 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.857169 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.857170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.857846 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.857847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.858385 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.858387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.858752 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.858753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.858822 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.859211 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.859215 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.860286 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.861617 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.861618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.861619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.861790 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.862026 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.862026 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.862027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.862082 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.862368 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.862369 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.862408 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.862491 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.862579 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 13:42:42.862697 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.856813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.19.53.253:41489)
0924 13:42:42.856874 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.856877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.856895 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:42:42.856915 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.856916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.856929 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.856992 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.857247 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.857762 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.858484 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.858684 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.858755 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.860373 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.860375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.860375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.860382 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.861534 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.861865 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.861866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.861866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.861972 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.862154 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.862155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.862155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.862287 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.862367 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.862514 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.862616 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 13:42:42.863104 26480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.252@KRBTEST.COM for oryx/127.19.53.252@KRBTEST.COM
I20260924 13:42:42.868750 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.863334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:36097 (local address 127.0.0.1:56854)
0924 13:42:42.863381 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.863391 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.863429 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.863542 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.863543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.863642 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.863666 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.863667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.864235 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.864236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.864704 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.864705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.865112 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.865113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.865149 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.865478 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.865481 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.866716 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.868036 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.868037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.868037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.868207 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.868352 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.868353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.868353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.868393 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.868544 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.868545 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.868574 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.868617 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.868680 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 13:42:42.868846 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.863408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.19.53.252:36097)
0924 13:42:42.863456 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.863458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.863472 (+ 14us) server_negotiation.cc:407] Connection header received
0924 13:42:42.863491 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.863491 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.863498 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.863540 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.863717 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.864149 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.864769 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.865042 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.865112 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.866790 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.866791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.866791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.866797 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.867965 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.868268 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.868269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.868270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.868313 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.868425 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.868426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.868426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.868510 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.868551 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.868633 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.868720 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":33}
I20260924 13:42:42.870154 26480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: Bootstrap starting.
I20260924 13:42:42.870960 26480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:42.871203 26480 log.cc:824] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:42.871847 26480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: No bootstrap required, opened a new log
I20260924 13:42:42.873333 26480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } }
I20260924 13:42:42.873445 26480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:42.873468 26480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68acfc076b894170aa626af998518c5a, State: Initialized, Role: FOLLOWER
I20260924 13:42:42.873584 26480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } }
I20260924 13:42:42.873976 26480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:42.873893 26485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } } }
I20260924 13:42:42.874032 26485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: This master's current role is: FOLLOWER
W20260924 13:42:42.875658 26496 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:42.875707 26496 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:42.877900 26412 master_runner.cc:428] Detected that this master 68acfc076b894170aa626af998518c5a is joining an existing cluster
I20260924 13:42:42.877964 26412 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.53.252:36097
I20260924 13:42:42.885318 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.879288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:36097 (local address 127.0.0.1:56858)
0924 13:42:42.879451 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.879459 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.879528 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.879668 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.879668 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.879854 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.879897 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.879898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.880461 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.880462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.881314 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.881317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.881788 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.881788 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.881836 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.882338 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.882343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.882659 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.884269 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.884270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.884270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.884486 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.884685 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.884685 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.884686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.884744 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.884970 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.884971 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.885007 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.885084 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.885197 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260924 13:42:42.885321 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.879337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.19.53.252:36097)
0924 13:42:42.879404 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.879406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.879511 (+ 105us) server_negotiation.cc:407] Connection header received
0924 13:42:42.879586 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.879587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.879593 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.879619 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.879966 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.880386 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.881399 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.881679 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.881714 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.882772 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.882773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.882774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.882780 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.884147 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.884565 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.884566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.884566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.884616 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.884794 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.884794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.884795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.884899 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.885005 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.885102 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.885245 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":50}
I20260924 13:42:42.905869 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.879184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48778 (local address 127.19.53.253:41489)
0924 13:42:42.879224 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.879226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.879466 (+ 240us) server_negotiation.cc:407] Connection header received
0924 13:42:42.879559 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.879561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.879570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.879600 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.880862 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.881600 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.882694 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.882995 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.883039 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.903071 (+ 20032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.903073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.903074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.903083 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.904615 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.905007 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.905008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.905009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.905060 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.905271 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.905272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.905272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.905427 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.905495 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.905673 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.905770 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:42:42.905879 26503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.879108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:41489 (local address 127.0.0.1:48778)
0924 13:42:42.879411 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.879420 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.879491 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.879659 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.879660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.880718 (+ 1058us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.880773 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.880775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.881694 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.881694 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.882596 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.882598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.883115 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.883115 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.883222 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.902431 (+ 19209us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.902437 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.902933 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.904715 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.904716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.904716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.904897 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.905136 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.905136 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.905137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.905195 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.905524 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.905525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.905573 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.905642 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.905756 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":64,"threads_started":1}
I20260924 13:42:42.906194 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.878897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.19.53.254:34111)
0924 13:42:42.878936 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.878937 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.879160 (+ 223us) server_negotiation.cc:407] Connection header received
0924 13:42:42.879249 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.879250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.879262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.879294 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.879758 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.880489 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.881472 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.881758 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.881797 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.903095 (+ 21298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.903097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.903098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.903107 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.904908 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.905328 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.905330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.905330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.905394 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.905592 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.905593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.905593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.905726 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.905790 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.906070 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.906135 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:42.906608 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.878876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:45550)
0924 13:42:42.879045 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.879055 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.879128 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.879415 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.879416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.879624 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.879699 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.879701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.880564 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.880565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.881397 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.881398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.881863 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.881864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.881920 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.902595 (+ 20675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.902600 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.902947 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.904999 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.904999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.905000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.905206 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.905461 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.905462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.905463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.905526 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.905817 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.905818 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.905860 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.906334 (+ 474us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.906470 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260924 13:42:42.907496 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 25823
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257361, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.254@KRBTEST.COM for oryx/127.19.53.252@KRBTEST.COM
I20260924 13:42:42.914676 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:39725
--local_ip_for_outbound_sockets=127.19.53.193
--tserver_master_addrs=127.19.53.254:34111
--webserver_port=42223
--webserver_interface=127.19.53.193
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab
--principal=oryx/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.53.254:34111,127.19.53.253:41489
--tserver_master_addrs=127.19.53.254:34111,127.19.53.253:41489,127.19.53.252:36097 with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:42:42.917814 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.910063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.19.53.252:36097)
0924 13:42:42.910122 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.910124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.910132 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:42.910166 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.910167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.910177 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.910230 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.910605 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.911369 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.912406 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.912745 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.912824 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.914908 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.914909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.914910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.914916 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.916532 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.916996 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.916997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.916998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.917048 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.917279 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.917280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.917280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.917403 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.917467 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.917655 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.917726 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 13:42:42.918166 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.909110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:36097 (local address 127.0.0.1:56874)
0924 13:42:42.909148 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.909158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.909194 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.910282 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.910283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.910460 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.910521 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.910523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.911459 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.911460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.912303 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.912305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.912832 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.912833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.912904 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.913334 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.913339 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.914812 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.916698 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.916699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.916700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.916901 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.917122 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.917124 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.917124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.917203 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.917474 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.917476 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.917528 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.917603 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.917714 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 13:42:42.918949 25761 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.53.252:36097: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "68acfc076b894170aa626af998518c5a"
member_type: NON_VOTER
last_known_addr {
host: "127.19.53.252"
port: 36097
}
attrs {
promote: true
}
}
dest_uuid: "8ee85e1707af4b85bd01c90c229805a2"
cas_config_opid_index: 5
I20260924 13:42:42.919240 25761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } }
I20260924 13:42:42.919636 26320 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 68acfc076b894170aa626af998518c5a (127.19.53.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } } }
I20260924 13:42:42.920606 26320 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.53.252:36097
I20260924 13:42:42.920686 26320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } } }
I20260924 13:42:42.920755 26320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:42.920990 26412 master.cc:537] Master@127.19.53.252:36097 shutting down...
W20260924 13:42:42.921363 25748 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 -> Peer 68acfc076b894170aa626af998518c5a (127.19.53.252:36097): Couldn't send request to peer 68acfc076b894170aa626af998518c5a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 13:42:42.925014 26412 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:42.925108 26412 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:42.925159 26412 tablet_replica.cc:330] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: stopping tablet replica
I20260924 13:42:42.926669 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.921002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.19.53.253:41489)
0924 13:42:42.921184 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.921186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.921196 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:42:42.921223 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.921224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.921250 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.921323 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:42.922542 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.923353 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.925534 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.926219 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.926271 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.926336 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.926562 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.926601 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260924 13:42:42.927002 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.919711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:41489 (local address 127.0.0.1:48786)
0924 13:42:42.920058 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.920069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.920125 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.922389 (+ 2264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.922391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.922393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:42.922436 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.922438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.923589 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.923590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.925423 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.925427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.926395 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.926395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.926459 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.926482 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.926897 (+ 415us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.926938 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328}
I20260924 13:42:42.927378 26368 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 LEARNER]: Refusing update from remote peer 8ee85e1707af4b85bd01c90c229805a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 13:42:42.927587 26319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 13:42:42.928148 26368 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 68acfc076b894170aa626af998518c5a (127.19.53.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } } }
I20260924 13:42:42.929222 26507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } } }
I20260924 13:42:42.929320 26507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: This master's current role is: LEARNER
I20260924 13:42:42.929464 26397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee85e1707af4b85bd01c90c229805a2. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: NON_VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: true } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } } }
I20260924 13:42:42.929543 26397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: This master's current role is: LEARNER
I20260924 13:42:42.929765 26319 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2: attempting to promote NON_VOTER 7e5fa5965fd84e7f964d462f6ae37efe to VOTER
I20260924 13:42:42.930126 26319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } }
I20260924 13:42:42.930564 26368 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 LEARNER]: Refusing update from remote peer 8ee85e1707af4b85bd01c90c229805a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 13:42:42.930770 26508 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 13:42:42.931113 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.930531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:36097 (local address 127.0.0.1:56886)
0924 13:42:42.930788 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.930833 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:36097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242}
I20260924 13:42:42.931507 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.931420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:36097 (local address 127.0.0.1:56890)
0924 13:42:42.931460 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.931487 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:36097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 13:42:42.931654 25748 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 -> Peer 68acfc076b894170aa626af998518c5a (127.19.53.252:36097): Couldn't send request to peer 68acfc076b894170aa626af998518c5a. Status: Network error: Client connection negotiation failed: client connection to 127.19.53.252:36097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 13:42:42.931731 26320 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7e5fa5965fd84e7f964d462f6ae37efe (127.19.53.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } } }
I20260924 13:42:42.931924 26368 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7e5fa5965fd84e7f964d462f6ae37efe (127.19.53.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } } }
I20260924 13:42:42.932639 26320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } } }
I20260924 13:42:42.932706 26320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:42.932749 26320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } } }
I20260924 13:42:42.932732 26397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } } }
I20260924 13:42:42.932787 26320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:42.932797 26397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:42.938356 26412 master.cc:552] Master@127.19.53.252:36097 shutdown complete.
I20260924 13:42:42.938413 26412 master_runner.cc:305] Clearing existing system tablet
I20260924 13:42:42.938555 26412 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:42.938601 26412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:42.939321 26412 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:42.939858 26511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.940011 26412 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:42.940070 26412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.940092 26412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/wal
uuid: "68acfc076b894170aa626af998518c5a"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "1137352e9cb96d81b7c3774e275c95a2"
server_key_iv: "3b2719703f61647b6a47738017c38258"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.940824 26412 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:42.942410 26412 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 13:42:42.942480 26412 log.cc:1196] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 13:42:42.942680 26412 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: Deleting consensus metadata
I20260924 13:42:42.942818 26412 master_runner.cc:315] Copying system tablet from 127.19.53.254:34111
I20260924 13:42:42.943396 26412 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.53.254:34111
I20260924 13:42:42.951025 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.943672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:45566)
0924 13:42:42.943788 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:42.943800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:42.943883 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:42.944031 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:42.944032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:42.944254 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:42.944320 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.944322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.945435 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.945436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.946436 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:42.946439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.946887 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.946888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:42.946964 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.947441 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:42.947446 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:42.947821 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:42.949757 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.949758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.949758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.950023 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.950245 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:42.950246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:42.950247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:42.950343 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:42.950632 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:42.950633 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:42.950679 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 13:42:42.950775 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 13:42:42.950921 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260924 13:42:42.951027 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:42.943742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.19.53.254:34111)
0924 13:42:42.943796 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:42.943799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:42.943870 (+ 71us) server_negotiation.cc:407] Connection header received
0924 13:42:42.943935 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:42.943936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:42.943947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:42.944022 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:42.944407 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.945179 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.946520 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:42.946804 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:42.946876 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:42.947936 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:42.947937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:42.947938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:42.947945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:42.949633 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.950112 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.950113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.950114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.950174 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:42.950423 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:42.950423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:42.950424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:42.950551 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:42.950620 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:42.950810 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 13:42:42.950905 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:42:42.951671 25781 tablet_copy_service.cc:140] P 8ee85e1707af4b85bd01c90c229805a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 68acfc076b894170aa626af998518c5a ({username='oryx', principal='oryx/127.19.53.252@KRBTEST.COM'} at 127.0.0.1:45566)
I20260924 13:42:42.951743 25781 tablet_copy_service.cc:161] P 8ee85e1707af4b85bd01c90c229805a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 68acfc076b894170aa626af998518c5a at {username='oryx', principal='oryx/127.19.53.252@KRBTEST.COM'} at 127.0.0.1:45566: session id = 68acfc076b894170aa626af998518c5a-00000000000000000000000000000000
I20260924 13:42:42.952335 25781 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 13:42:42.952769 26412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 13:42:42.954094 26412 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: tablet copy: Starting download of 0 data blocks...
I20260924 13:42:42.954246 26412 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: tablet copy: Starting download of 1 WAL segments...
I20260924 13:42:42.954980 26412 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 13:42:42.955791 25781 tablet_copy_service.cc:342] P 8ee85e1707af4b85bd01c90c229805a2: Request end of tablet copy session 68acfc076b894170aa626af998518c5a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.53.252@KRBTEST.COM'} at 127.0.0.1:45566
I20260924 13:42:42.955859 25781 tablet_copy_service.cc:433] P 8ee85e1707af4b85bd01c90c229805a2: ending tablet copy session 68acfc076b894170aa626af998518c5a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 68acfc076b894170aa626af998518c5a
I20260924 13:42:42.958387 26412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:42.959743 26521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:42.959784 26518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:42.959794 26519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:42.960361 26412 server_base.cc:1034] running on GCE node
I20260924 13:42:42.960460 26412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:42.960599 26412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:42.961745 26412 hybrid_clock.cc:648] HybridClock initialized: now 1790257362961738 us; error 24 us; skew 500 ppm
Sep 24 13:42:42 dist-test-slave-tfm9 krb5kdc[25716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257362, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:42.963279 26412 init.cc:377] Logged in from keytab as oryx/127.19.53.252@KRBTEST.COM (short username oryx)
I20260924 13:42:42.963543 26412 webserver.cc:460] Webserver started at http://127.19.53.252:32965/ using document root <none> and password file <none>
I20260924 13:42:42.963660 26412 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:42.963696 26412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:42.964480 26412 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:42.965126 26528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:42.965310 26412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:42.965365 26412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/wal
uuid: "68acfc076b894170aa626af998518c5a"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "1137352e9cb96d81b7c3774e275c95a2"
server_key_iv: "3b2719703f61647b6a47738017c38258"
server_key_version: "encryptionkey@0"
I20260924 13:42:42.965425 26412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:42.985517 26412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:42.987434 26412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:42.990535 26412 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.252:36097
I20260924 13:42:42.990886 26412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/master-2/data/info.pb
I20260924 13:42:42.991775 26581 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:42:42.992300 26581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: Bootstrap starting.
I20260924 13:42:42.993556 26580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.252:36097 every 8 connection(s)
I20260924 13:42:42.995195 26581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:42:42.995412 26581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: Bootstrap complete.
I20260924 13:42:42.995644 26581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } }
I20260924 13:42:42.995733 26581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 68acfc076b894170aa626af998518c5a, State: Initialized, Role: LEARNER
I20260924 13:42:42.995770 26581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } }
I20260924 13:42:42.996002 26581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:42.995982 26584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } } }
I20260924 13:42:42.996241 26584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: This master's current role is: LEARNER
I20260924 13:42:42.997465 26595 catalog_manager.cc:1259] Loaded cluster ID: 2e5a03513b4546cd9964bbd188ba433d
I20260924 13:42:42.997548 26595 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: loading cluster ID for follower catalog manager: success
I20260924 13:42:42.998836 26595 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: acquiring CA information for follower catalog manager: success
I20260924 13:42:42.999166 26595 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 13:42:43.014546 26505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:43.014750 26505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:43.014775 26505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:43.015952 26505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:43.015995 26505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:43.016021 26505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:43.017513 26505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab
--principal=oryx/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:39725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=42223
--tserver_master_addrs=127.19.53.254:34111,127.19.53.253:41489,127.19.53.252:36097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:43.017768 26505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:43.018045 26505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:43.023710 26601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:43.023726 26603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:43.023718 26600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:43.024317 26505 server_base.cc:1034] running on GCE node
I20260924 13:42:43.024508 26505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:43.024757 26505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:43.025955 26505 hybrid_clock.cc:648] HybridClock initialized: now 1790257363025916 us; error 54 us; skew 500 ppm
Sep 24 13:42:43 dist-test-slave-tfm9 krb5kdc[25716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257363, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:43.029006 26505 init.cc:377] Logged in from keytab as oryx/127.19.53.193@KRBTEST.COM (short username oryx)
I20260924 13:42:43.029448 26505 webserver.cc:460] Webserver started at http://127.19.53.193:42223/ using document root <none> and password file <none>
I20260924 13:42:43.029716 26505 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:43.029803 26505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:43.031267 26505 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:43.031958 26610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:43.032151 26505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:43.032261 26505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "aded32df8cd54b0d9b2861430781094e"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "15836c72c31cc8ac15c9303772dfbfd3"
server_key_iv: "01dc8be526e05f52f1b0d2c427a26cc7"
server_key_version: "encryptionkey@0"
I20260924 13:42:43.032626 26505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:43.047159 26505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:43.049616 26505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:43.049782 26505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:43.050089 26505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:43.050448 26505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:43.050505 26505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:43.050540 26505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:43.050567 26505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:43.056150 26505 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:39725
I20260924 13:42:43.056262 26723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:39725 every 8 connection(s)
I20260924 13:42:43.056576 26505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:42:43.060649 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 26505
I20260924 13:42:43.060814 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 25957
Sep 24 13:42:43 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257363, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.252@KRBTEST.COM
Sep 24 13:42:43 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257363, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.253@KRBTEST.COM
Sep 24 13:42:43 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257363, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:43.069481 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:37865
--local_ip_for_outbound_sockets=127.19.53.194
--tserver_master_addrs=127.19.53.254:34111
--webserver_port=46235
--webserver_interface=127.19.53.194
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab
--principal=oryx/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.53.254:34111,127.19.53.253:41489
--tserver_master_addrs=127.19.53.254:34111,127.19.53.253:41489,127.19.53.252:36097 with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:42:43.069841 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.058244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:54157 (local address 127.19.53.253:41489)
0924 13:42:43.058297 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.058299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.058353 (+ 54us) server_negotiation.cc:407] Connection header received
0924 13:42:43.059065 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.059066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.059079 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.059131 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.059869 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.060623 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.062209 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.062493 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.062532 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.066802 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.066803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.066803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.066810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.068298 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.068793 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.068794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.068794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.068853 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.069091 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.069093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.069093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.069257 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.069298 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.069646 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.069731 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:42:43.069880 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.057542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:55795 (local address 127.19.53.254:34111)
0924 13:42:43.057585 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.057587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.057680 (+ 93us) server_negotiation.cc:407] Connection header received
0924 13:42:43.058992 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.058993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.059007 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.059073 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.059858 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.060381 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.062006 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.062347 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.062389 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.066521 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.066522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.066522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.066529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.068166 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.068748 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.068749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.068750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.068806 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.069078 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.069079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.069079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.069218 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.069271 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.069651 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.069738 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:43.070442 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.058173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:41489 (local address 127.19.53.193:54157)
0924 13:42:43.058295 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.058306 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.058887 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.059152 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.059156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.059589 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:43.059748 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.059755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.060784 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.060785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.062126 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.062128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.062618 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.062619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.062676 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.063418 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:43.063422 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:43.066699 (+ 3277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:43.068405 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.068406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.068406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.068701 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.068929 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.068930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.068931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.069013 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.069347 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:43.069348 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:43.069480 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.069958 (+ 478us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.070180 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260924 13:42:43.070444 26729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.057304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.193:55795)
0924 13:42:43.057608 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.057616 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.058877 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.059148 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.059153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.059544 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:43.059747 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.059752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.060498 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.060499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.061883 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.061886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.062478 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.062479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.062572 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.063248 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:43.063259 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:43.066411 (+ 3152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:43.068253 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.068259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.068261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.068667 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.068870 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.068871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.068871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.069005 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.069308 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:43.069314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:43.069484 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.069978 (+ 494us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.070269 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":131,"spinlock_wait_cycles":6912,"thread_start_us":92,"threads_started":1}
I20260924 13:42:43.070963 26725 heartbeater.cc:347] Connected to a master server at 127.19.53.254:34111
I20260924 13:42:43.071086 26725 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:43.071091 26730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.057304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:36097 (local address 127.19.53.193:54707)
0924 13:42:43.057556 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.057605 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.058878 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.059293 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.059294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.059593 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:43.059748 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.059755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.060929 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.060930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.061878 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.061881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.062420 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.062421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.062562 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.063287 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:43.063293 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:43.065368 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:43.068949 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.068949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.068950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.069169 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.069375 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.069376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.069376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.069428 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.070317 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:43.070318 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:43.070360 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.070407 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.070919 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":19,"thread_start_us":86,"threads_started":1}
I20260924 13:42:43.071296 26726 heartbeater.cc:347] Connected to a master server at 127.19.53.253:41489
I20260924 13:42:43.071298 26725 heartbeater.cc:511] Master 127.19.53.254:34111 requested a full tablet report, sending...
I20260924 13:42:43.071347 26726 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:43.071431 26726 heartbeater.cc:511] Master 127.19.53.253:41489 requested a full tablet report, sending...
I20260924 13:42:43.071691 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.057329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:54707 (local address 127.19.53.252:36097)
0924 13:42:43.057445 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.057447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.057693 (+ 246us) server_negotiation.cc:407] Connection header received
0924 13:42:43.059191 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.059192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.059207 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.059276 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.059860 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.060851 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.062012 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.062336 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.062792 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.065525 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.065526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.065526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.065535 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.068865 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.069248 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.069249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.069250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.069306 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.069496 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.069496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.069497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.069867 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.071020 (+ 1153us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.071112 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.071568 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260924 13:42:43.071795 25761 ts_manager.cc:194] Re-registered known tserver with Master: aded32df8cd54b0d9b2861430781094e (127.19.53.193:39725)
I20260924 13:42:43.071870 26358 ts_manager.cc:194] Registered new tserver with Master: aded32df8cd54b0d9b2861430781094e (127.19.53.193:39725)
I20260924 13:42:43.072033 26724 heartbeater.cc:347] Connected to a master server at 127.19.53.252:36097
I20260924 13:42:43.072077 26724 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:43.072132 26724 heartbeater.cc:511] Master 127.19.53.252:36097 requested a full tablet report, sending...
I20260924 13:42:43.072286 25761 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:55795
I20260924 13:42:43.072461 26545 ts_manager.cc:194] Registered new tserver with Master: aded32df8cd54b0d9b2861430781094e (127.19.53.193:39725)
W20260924 13:42:43.152817 26732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:43.153040 26732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:43.153077 26732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:43.154309 26732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:43.154352 26732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:43.154379 26732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:43.155802 26732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab
--principal=oryx/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:37865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=46235
--tserver_master_addrs=127.19.53.254:34111,127.19.53.253:41489,127.19.53.252:36097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:43.155983 26732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:43.156167 26732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:43.161325 26737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:43.161334 26738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:43.161348 26740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:43.161504 26732 server_base.cc:1034] running on GCE node
I20260924 13:42:43.161751 26732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:43.161990 26732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:43.163138 26732 hybrid_clock.cc:648] HybridClock initialized: now 1790257363163104 us; error 51 us; skew 500 ppm
Sep 24 13:42:43 dist-test-slave-tfm9 krb5kdc[25716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257363, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:43.165617 26732 init.cc:377] Logged in from keytab as oryx/127.19.53.194@KRBTEST.COM (short username oryx)
I20260924 13:42:43.166127 26732 webserver.cc:460] Webserver started at http://127.19.53.194:46235/ using document root <none> and password file <none>
I20260924 13:42:43.166422 26732 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:43.166504 26732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:43.167820 26732 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 13:42:43.168387 26747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:43.168555 26732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:43.168658 26732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "8de5b4cde81d40c3bcee0982b73f5573"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "2de85aa9d4cfa1ad805c401eebc381de"
server_key_iv: "b1e0cd12ea65fb4769908d94e68dd3a3"
server_key_version: "encryptionkey@0"
I20260924 13:42:43.168993 26732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:43.190183 26732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:43.192579 26732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:43.192742 26732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:43.192966 26732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:43.193284 26732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:43.193336 26732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:43.193377 26732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:43.193403 26732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:43.199689 26732 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:37865
I20260924 13:42:43.199741 26862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:37865 every 8 connection(s)
I20260924 13:42:43.200115 26732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:42:43.205899 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 26732
I20260924 13:42:43.206106 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 26091
Sep 24 13:42:43 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257363, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.252@KRBTEST.COM
Sep 24 13:42:43 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257363, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.253@KRBTEST.COM
Sep 24 13:42:43 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257363, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:43.211992 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.201265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:36509 (local address 127.19.53.253:41489)
0924 13:42:43.201316 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.201318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.201365 (+ 47us) server_negotiation.cc:407] Connection header received
0924 13:42:43.201851 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.201852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.201866 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.201898 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.202759 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.203491 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.204739 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.205040 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.205112 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.208936 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.208937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.208937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.208945 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.210544 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.211077 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.211078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.211079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.211140 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.211342 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.211343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.211343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.211475 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.211514 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.211803 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.211898 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:42:43.211992 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.200864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:57483 (local address 127.19.53.252:36097)
0924 13:42:43.200962 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.200966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.201148 (+ 182us) server_negotiation.cc:407] Connection header received
0924 13:42:43.201840 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.201841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.201853 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.201924 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.202722 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.203252 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.204742 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.205055 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.205112 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.208502 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.208503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.208504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.208511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.210238 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.210877 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.210878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.210878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.210940 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.211231 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.211232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.211233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.211372 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.211414 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.211803 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.211900 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 13:42:43.212621 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.201193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:41489 (local address 127.19.53.194:36509)
0924 13:42:43.201309 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.201319 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.201728 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.202007 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.202011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.202582 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:43.202646 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.202648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.203590 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.203590 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.204648 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.204650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.205139 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.205140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.205272 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.205945 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:43.205955 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:43.208833 (+ 2878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:43.210627 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.210628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.210629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.210994 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.211217 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.211217 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.211218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.211273 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.211556 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:43.211557 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:43.211628 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.212146 (+ 518us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.212374 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":9,"thread_start_us":64,"threads_started":1}
I20260924 13:42:43.212621 26867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.200853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:36097 (local address 127.19.53.194:57483)
0924 13:42:43.201016 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.201051 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.201727 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.202007 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.202011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.202396 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:43.202590 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.202596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.203349 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.203349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.204648 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.204650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.205168 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.205169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.205268 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.206683 (+ 1415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:43.206688 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:43.208389 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:43.210330 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.210337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.210340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.210772 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.211016 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.211016 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.211017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.211150 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.211458 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:43.211464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:43.211628 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.212146 (+ 518us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.212420 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":170,"thread_start_us":99,"threads_started":1}
I20260924 13:42:43.213187 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:35269
--local_ip_for_outbound_sockets=127.19.53.195
--tserver_master_addrs=127.19.53.254:34111
--webserver_port=44979
--webserver_interface=127.19.53.195
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab
--principal=oryx/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.53.254:34111,127.19.53.253:41489
--tserver_master_addrs=127.19.53.254:34111,127.19.53.253:41489,127.19.53.252:36097 with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:42:43.213654 26863 heartbeater.cc:347] Connected to a master server at 127.19.53.252:36097
I20260924 13:42:43.213752 26863 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:43.213959 26863 heartbeater.cc:511] Master 127.19.53.252:36097 requested a full tablet report, sending...
I20260924 13:42:43.214439 26545 ts_manager.cc:194] Registered new tserver with Master: 8de5b4cde81d40c3bcee0982b73f5573 (127.19.53.194:37865)
I20260924 13:42:43.214771 26865 heartbeater.cc:347] Connected to a master server at 127.19.53.253:41489
I20260924 13:42:43.214809 26865 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:43.214872 26865 heartbeater.cc:511] Master 127.19.53.253:41489 requested a full tablet report, sending...
I20260924 13:42:43.215162 26358 ts_manager.cc:194] Registered new tserver with Master: 8de5b4cde81d40c3bcee0982b73f5573 (127.19.53.194:37865)
I20260924 13:42:43.216338 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.201560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:47365 (local address 127.19.53.254:34111)
0924 13:42:43.201607 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.201610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.201625 (+ 15us) server_negotiation.cc:407] Connection header received
0924 13:42:43.201977 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.201979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.201989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.202054 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.202684 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.203491 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.205517 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.205823 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.205889 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.212685 (+ 6796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.212687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.212688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.212696 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.215165 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.215582 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.215583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.215583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.215641 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.215828 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.215829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.215829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.215944 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.216026 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.216178 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.216251 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:42:43.216355 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.201373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.194:47365)
0924 13:42:43.201473 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.201485 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.201746 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.202073 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.202074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.202389 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:43.202589 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.202593 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.204748 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.204749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.205400 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.205402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.205963 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.205964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.206028 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.210540 (+ 4512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:43.210545 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:43.212581 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:43.215312 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.215313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.215313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.215503 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.215701 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.215702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.215702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.215754 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.216017 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:43.216018 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:43.216059 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.216148 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.216251 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":57,"threads_started":1}
I20260924 13:42:43.216635 26864 heartbeater.cc:347] Connected to a master server at 127.19.53.254:34111
I20260924 13:42:43.216706 26864 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:43.216777 26864 heartbeater.cc:511] Master 127.19.53.254:34111 requested a full tablet report, sending...
I20260924 13:42:43.216986 25761 ts_manager.cc:194] Re-registered known tserver with Master: 8de5b4cde81d40c3bcee0982b73f5573 (127.19.53.194:37865)
I20260924 13:42:43.217360 25761 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:47365
W20260924 13:42:43.295378 26870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:43.295571 26870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:43.295596 26870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:43.296823 26870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:43.296862 26870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:43.296888 26870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:43.298467 26870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab
--principal=oryx/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:35269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=44979
--tserver_master_addrs=127.19.53.254:34111,127.19.53.253:41489,127.19.53.252:36097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:43.298698 26870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:43.298938 26870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:43.303833 26876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:43.303861 26875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:43.303876 26870 server_base.cc:1034] running on GCE node
W20260924 13:42:43.303843 26878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:43.304212 26870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:43.304486 26870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:43.305634 26870 hybrid_clock.cc:648] HybridClock initialized: now 1790257363305621 us; error 28 us; skew 500 ppm
Sep 24 13:42:43 dist-test-slave-tfm9 krb5kdc[25716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257363, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:43.308118 26870 init.cc:377] Logged in from keytab as oryx/127.19.53.195@KRBTEST.COM (short username oryx)
I20260924 13:42:43.308535 26870 webserver.cc:460] Webserver started at http://127.19.53.195:44979/ using document root <none> and password file <none>
I20260924 13:42:43.308753 26870 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:43.308804 26870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:43.310187 26870 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:43.310784 26885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:43.310974 26870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:43.311062 26870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "3e51301c17dd4526937621aeba3fed60"
format_stamp: "Formatted at 2026-09-24 13:42:42 on dist-test-slave-tfm9"
server_key: "150f32fc75057340b9929303f960e3f6"
server_key_iv: "6793bef67712ceb0e871ffcba069efa0"
server_key_version: "encryptionkey@0"
I20260924 13:42:43.311386 26870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:43.333581 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.330528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.19.53.252:36097)
0924 13:42:43.330584 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.330587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.330601 (+ 14us) server_negotiation.cc:407] Connection header received
0924 13:42:43.330626 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.330627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.330645 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.330714 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:43.330842 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.331573 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.332899 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.333298 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.333360 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.333412 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.333500 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.333523 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 13:42:43.333608 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.330454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:36097 (local address 127.0.0.1:56904)
0924 13:42:43.330505 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.330515 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.330553 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.330732 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.330733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.330734 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:43.330782 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.330784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.331670 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.331671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.332811 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.332814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.333376 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.333377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.333438 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.333442 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.333495 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.333533 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 13:42:43.334206 26552 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 13:42:43.334367 26584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee85e1707af4b85bd01c90c229805a2. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } } }
I20260924 13:42:43.334466 26584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: This master's current role is: LEARNER
I20260924 13:42:43.343127 26870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:43.344944 26870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:43.345098 26870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:43.345340 26870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:43.345705 26870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:43.345750 26870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:43.345783 26870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:43.345806 26870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:43.351601 26870 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:35269
I20260924 13:42:43.351644 26999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:35269 every 8 connection(s)
I20260924 13:42:43.351980 26870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:42:43.358592 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 26870
Sep 24 13:42:43 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257363, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
Sep 24 13:42:43 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257363, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.253@KRBTEST.COM
Sep 24 13:42:43 dist-test-slave-tfm9 krb5kdc[25716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257363, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.252@KRBTEST.COM
I20260924 13:42:43.363844 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.352733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:57479 (local address 127.19.53.253:41489)
0924 13:42:43.352772 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.352774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.353216 (+ 442us) server_negotiation.cc:407] Connection header received
0924 13:42:43.354183 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.354184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.354197 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.354229 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.354954 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.355700 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.356844 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.357132 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.357196 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.360423 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.360424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.360424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.360431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.362248 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.362786 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.362787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.362788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.362849 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.363155 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.363156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.363157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.363285 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.363324 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.363683 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.363755 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 13:42:43.364262 27004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.352680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:36097 (local address 127.19.53.195:41513)
0924 13:42:43.352866 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.352918 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.354061 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.354295 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.354299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.354652 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:43.354847 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.354851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.355538 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.355539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.356547 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.356550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.356960 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.356961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.357112 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.357962 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:43.357967 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:43.360615 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:43.362897 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.362898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.362898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.363117 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.363343 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.363343 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.363344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.363416 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.363694 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:43.363695 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:43.363738 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.363825 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.364008 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
I20260924 13:42:43.364265 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.352679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:41489 (local address 127.19.53.195:57479)
0924 13:42:43.353133 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.353144 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.354081 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.354311 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.354312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.354708 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:43.354848 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.354855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.355803 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.355804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.356739 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.356741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.357251 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.357252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.357320 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.357807 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:43.357813 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:43.360166 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:43.362347 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.362354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.362356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.362695 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.362928 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.362929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.362930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.363073 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.363368 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:43.363374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:43.363540 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.363920 (+ 380us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.364066 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":90,"thread_start_us":106,"threads_started":1}
I20260924 13:42:43.364449 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.353082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:54099 (local address 127.19.53.254:34111)
0924 13:42:43.353123 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.353125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.353239 (+ 114us) server_negotiation.cc:407] Connection header received
0924 13:42:43.354182 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.354184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.354196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.354229 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.354954 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.355719 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.356819 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.357107 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.357183 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.361141 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.361142 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.361142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.361149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.363313 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.363652 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.363653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.363654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.363705 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.363904 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.363905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.363905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.364029 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.364080 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.364269 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.364343 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:43.364718 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.352665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:41513 (local address 127.19.53.252:36097)
0924 13:42:43.352706 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.352709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.353012 (+ 303us) server_negotiation.cc:407] Connection header received
0924 13:42:43.354159 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.354160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.354172 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.354226 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.354948 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.355430 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.356635 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.356879 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.356944 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.360717 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.360719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.360719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.360727 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.362691 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.363200 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.363202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.363202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.363268 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.363483 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.363484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.363484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.363614 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.363670 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.364552 (+ 882us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.364629 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:42:43.364763 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.353031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.195:54099)
0924 13:42:43.353185 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.353193 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.354081 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.354311 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.354312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.354704 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:43.354848 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.354855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.355803 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.355804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.356741 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.356742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.357195 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.357196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.357260 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.357651 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:43.357662 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:43.361040 (+ 3378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:43.363387 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.363387 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.363388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.363573 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.363775 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:43.363776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:43.363777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:43.363826 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:43.364129 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:43.364129 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:43.364166 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.364552 (+ 386us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.364642 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":58,"thread_start_us":60,"threads_started":1}
I20260924 13:42:43.365140 27000 heartbeater.cc:347] Connected to a master server at 127.19.53.252:36097
I20260924 13:42:43.365242 27001 heartbeater.cc:347] Connected to a master server at 127.19.53.254:34111
I20260924 13:42:43.365253 27000 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:43.365324 27001 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:43.365406 27002 heartbeater.cc:347] Connected to a master server at 127.19.53.253:41489
I20260924 13:42:43.365489 27000 heartbeater.cc:511] Master 127.19.53.252:36097 requested a full tablet report, sending...
I20260924 13:42:43.365489 27001 heartbeater.cc:511] Master 127.19.53.254:34111 requested a full tablet report, sending...
I20260924 13:42:43.365492 27002 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:43.365723 27002 heartbeater.cc:511] Master 127.19.53.253:41489 requested a full tablet report, sending...
I20260924 13:42:43.365942 25761 ts_manager.cc:194] Re-registered known tserver with Master: 3e51301c17dd4526937621aeba3fed60 (127.19.53.195:35269)
I20260924 13:42:43.365958 26545 ts_manager.cc:194] Registered new tserver with Master: 3e51301c17dd4526937621aeba3fed60 (127.19.53.195:35269)
I20260924 13:42:43.366022 26358 ts_manager.cc:194] Registered new tserver with Master: 3e51301c17dd4526937621aeba3fed60 (127.19.53.195:35269)
I20260924 13:42:43.366407 25761 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:54099
I20260924 13:42:43.372566 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.360194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.19.53.253:41489)
0924 13:42:43.360575 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.360577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.362321 (+ 1744us) server_negotiation.cc:407] Connection header received
0924 13:42:43.362404 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.362405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.362418 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.362470 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.363225 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.364191 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.366475 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.366762 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.366829 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.369220 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.369221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.369221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.369226 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.371104 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.371664 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.371665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.371665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.371729 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.371922 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.371923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.371924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.372063 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.372162 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.372385 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.372471 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":55,"threads_started":1}
I20260924 13:42:43.373018 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.359695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45572 (local address 127.19.53.254:34111)
0924 13:42:43.359846 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.359848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.360005 (+ 157us) server_negotiation.cc:407] Connection header received
0924 13:42:43.360122 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.360123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.360134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.360177 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.361008 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.361904 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.367507 (+ 5603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.367709 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.367788 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.369002 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.369003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.369004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.369009 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.371518 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.372004 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.372005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.372005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.372065 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.372267 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.372268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.372268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.372413 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.372501 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.372862 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.372942 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260924 13:42:43.373559 27012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.359700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56908 (local address 127.19.53.252:36097)
0924 13:42:43.360618 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.360620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.366422 (+ 5802us) server_negotiation.cc:407] Connection header received
0924 13:42:43.366511 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.366512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.366524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.366562 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.367290 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.367742 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.368830 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.369128 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.369176 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.370627 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.370628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.370629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.370635 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.372294 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.372706 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.372708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.372708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.372780 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.372969 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.372969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.372970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.373111 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.373181 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.373379 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.373454 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":48,"threads_started":1}
I20260924 13:42:43.374578 25761 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:43.376271 25761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:43.376945 26319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: NON_VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: true } } }
I20260924 13:42:43.377086 26319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:43.378305 26320 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2: attempting to promote NON_VOTER 68acfc076b894170aa626af998518c5a to VOTER
I20260924 13:42:43.378422 26320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } }
I20260924 13:42:43.379042 26368 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 FOLLOWER]: Refusing update from remote peer 8ee85e1707af4b85bd01c90c229805a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 13:42:43.379094 26552 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 1 LEARNER]: Refusing update from remote peer 8ee85e1707af4b85bd01c90c229805a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 13:42:43.379241 26319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 13:42:43.379328 26508 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 13:42:43.379979 26319 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 68acfc076b894170aa626af998518c5a (127.19.53.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } } }
I20260924 13:42:43.380172 26368 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 68acfc076b894170aa626af998518c5a (127.19.53.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } } }
I20260924 13:42:43.380213 26552 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 68acfc076b894170aa626af998518c5a (127.19.53.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } } }
I20260924 13:42:43.380954 26319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } } }
I20260924 13:42:43.381038 26319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:43.381096 26319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } } }
I20260924 13:42:43.381148 26319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:43.381184 26319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } } }
I20260924 13:42:43.381143 26397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } } }
I20260924 13:42:43.381219 26319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:43.381225 26397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:43.381289 26584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ee85e1707af4b85bd01c90c229805a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } } }
I20260924 13:42:43.381357 26584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:43.385670 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.381540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39725 (local address 127.0.0.1:54442)
0924 13:42:43.381759 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.381770 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.381814 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.382093 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.382094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.382096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:43.382145 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.382147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.383473 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.383474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.384622 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.384624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.385466 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.385466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.385537 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.385541 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.385569 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.385605 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":42}
I20260924 13:42:43.386389 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.381706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.19.53.193:39725)
0924 13:42:43.381866 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.381871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.381881 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:42:43.381955 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.381957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.381974 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.382219 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:43.382283 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.383397 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.384693 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.385364 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.385843 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.386227 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.386255 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.386295 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260924 13:42:43.386462 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.381644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:35269 (local address 127.0.0.1:59884)
0924 13:42:43.382180 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.382190 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.382254 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.382430 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.382431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.382433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:43.382475 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.382477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.383887 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.383888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.385045 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.385047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.386238 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.386239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.386299 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.386303 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.386353 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.386392 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":80,"threads_started":1}
I20260924 13:42:43.386719 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.381723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.19.53.195:35269)
0924 13:42:43.382126 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.382131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.382247 (+ 116us) server_negotiation.cc:407] Connection header received
0924 13:42:43.382322 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.382324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.382339 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.382419 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:43.382534 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.383814 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.385122 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.386043 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.386116 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.386199 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.386589 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.386629 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":86,"threads_started":1}
I20260924 13:42:43.386770 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.381969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:37865 (local address 127.0.0.1:33374)
0924 13:42:43.382245 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.382254 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.382294 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.382589 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.382590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.382591 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:43.382619 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.382620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.383448 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.383450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.385603 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.385605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.386371 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.386372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.386565 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.386570 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.386619 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.386658 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":1105,"thread_start_us":64,"threads_started":1}
I20260924 13:42:43.387076 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.382083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33374 (local address 127.19.53.194:37865)
0924 13:42:43.382432 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.382437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.382446 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:43.382493 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.382495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.382512 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.382593 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:43.382676 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.383316 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.385680 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.386268 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.386365 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.386432 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.386939 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.386997 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":81,"threads_started":1}
I20260924 13:42:43.387075 26658 tablet_service.cc:1467] Processing CreateTablet for tablet 508b215ff77a4bda9ac6e06f6c827918 (DEFAULT_TABLE table=test-table [id=a3dbf6d5d00d4b63a0e5e7a4d726b09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:43.387249 26934 tablet_service.cc:1467] Processing CreateTablet for tablet 508b215ff77a4bda9ac6e06f6c827918 (DEFAULT_TABLE table=test-table [id=a3dbf6d5d00d4b63a0e5e7a4d726b09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:43.387418 26658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508b215ff77a4bda9ac6e06f6c827918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:43.387543 26934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508b215ff77a4bda9ac6e06f6c827918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:43.390277 26796 tablet_service.cc:1467] Processing CreateTablet for tablet 508b215ff77a4bda9ac6e06f6c827918 (DEFAULT_TABLE table=test-table [id=a3dbf6d5d00d4b63a0e5e7a4d726b09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:43.390596 26796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508b215ff77a4bda9ac6e06f6c827918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:43.390911 27030 tablet_bootstrap.cc:492] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60: Bootstrap starting.
I20260924 13:42:43.391608 27030 tablet_bootstrap.cc:654] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:43.391961 27030 log.cc:824] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:43.392306 27029 tablet_bootstrap.cc:492] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e: Bootstrap starting.
I20260924 13:42:43.392675 27030 tablet_bootstrap.cc:492] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60: No bootstrap required, opened a new log
I20260924 13:42:43.392755 27030 ts_tablet_manager.cc:1397] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:43.392920 27029 tablet_bootstrap.cc:654] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:43.393249 27029 log.cc:824] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:43.394086 27029 tablet_bootstrap.cc:492] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e: No bootstrap required, opened a new log
I20260924 13:42:43.394244 27034 tablet_bootstrap.cc:492] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573: Bootstrap starting.
I20260924 13:42:43.394286 27029 ts_tablet_manager.cc:1397] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:42:43.394940 27034 tablet_bootstrap.cc:654] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:43.395198 27030 raft_consensus.cc:348] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } } peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } }
I20260924 13:42:43.395354 27034 log.cc:824] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:43.395409 27030 raft_consensus.cc:374] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:43.395452 27030 raft_consensus.cc:729] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e51301c17dd4526937621aeba3fed60, State: Initialized, Role: FOLLOWER
I20260924 13:42:43.395563 27030 consensus_queue.cc:260] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } } peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } }
I20260924 13:42:43.395685 27029 raft_consensus.cc:348] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } } peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } }
I20260924 13:42:43.395859 27030 ts_tablet_manager.cc:1428] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:42:43.395901 27029 raft_consensus.cc:374] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:43.395948 27029 raft_consensus.cc:729] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aded32df8cd54b0d9b2861430781094e, State: Initialized, Role: FOLLOWER
I20260924 13:42:43.396049 27001 heartbeater.cc:503] Master 127.19.53.254:34111 was elected leader, sending a full tablet report...
I20260924 13:42:43.396098 27029 consensus_queue.cc:260] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } } peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } }
I20260924 13:42:43.396327 27029 ts_tablet_manager.cc:1428] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:42:43.396585 27034 tablet_bootstrap.cc:492] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573: No bootstrap required, opened a new log
I20260924 13:42:43.396637 27034 ts_tablet_manager.cc:1397] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:43.397671 26725 heartbeater.cc:503] Master 127.19.53.254:34111 was elected leader, sending a full tablet report...
I20260924 13:42:43.399260 27034 raft_consensus.cc:348] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } } peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } }
I20260924 13:42:43.399372 27034 raft_consensus.cc:374] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:43.399392 27034 raft_consensus.cc:729] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de5b4cde81d40c3bcee0982b73f5573, State: Initialized, Role: FOLLOWER
I20260924 13:42:43.399501 27034 consensus_queue.cc:260] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } } peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } }
I20260924 13:42:43.399703 27034 ts_tablet_manager.cc:1428] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:42:43.399814 26864 heartbeater.cc:503] Master 127.19.53.254:34111 was elected leader, sending a full tablet report...
W20260924 13:42:43.450893 26866 tablet.cc:2369] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:42:43.558537 26727 tablet.cc:2369] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:42:43.568328 27036 raft_consensus.cc:482] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:43.568460 27036 raft_consensus.cc:504] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } } peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } }
I20260924 13:42:43.568852 27036 leader_election.cc:288] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aded32df8cd54b0d9b2861430781094e (127.19.53.193:39725), 8de5b4cde81d40c3bcee0982b73f5573 (127.19.53.194:37865)
I20260924 13:42:43.572003 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.569178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:60507 (local address 127.19.53.193:39725)
0924 13:42:43.569223 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.569225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.569233 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:43.569260 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.569260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.569271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.569343 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:43.569518 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.570061 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.571103 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.571653 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.571730 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.571738 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.571886 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.571927 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:42:43.571997 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.569073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39725 (local address 127.19.53.195:60507)
0924 13:42:43.569110 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.569121 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.569168 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.569366 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.569368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.569369 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:43.569429 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.569430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.570138 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.570139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.570999 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.571003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.571734 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.571735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.571800 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.571804 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.571879 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.571919 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":43}
I20260924 13:42:43.572050 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.569177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:44951 (local address 127.19.53.194:37865)
0924 13:42:43.569219 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.569221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.569229 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:43.569263 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.569263 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.569270 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.569312 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:43.569518 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.570060 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.571075 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.571654 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.571730 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.571738 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.571942 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.571982 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:43.572103 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.569047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:37865 (local address 127.19.53.195:44951)
0924 13:42:43.569111 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:43.569124 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:43.569184 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:43.569333 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:43.569334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:43.569336 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:43.569413 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.569415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.570140 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.570141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.570949 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:43.570951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.571788 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.571789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:43.571848 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.571850 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:42:43.571932 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 13:42:43.571973 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":24}
I20260924 13:42:43.572438 26678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508b215ff77a4bda9ac6e06f6c827918" candidate_uuid: "3e51301c17dd4526937621aeba3fed60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aded32df8cd54b0d9b2861430781094e" is_pre_election: true
I20260924 13:42:43.572446 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508b215ff77a4bda9ac6e06f6c827918" candidate_uuid: "3e51301c17dd4526937621aeba3fed60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5b4cde81d40c3bcee0982b73f5573" is_pre_election: true
I20260924 13:42:43.572613 26678 raft_consensus.cc:2454] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e51301c17dd4526937621aeba3fed60 in term 0.
I20260924 13:42:43.572628 26816 raft_consensus.cc:2454] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e51301c17dd4526937621aeba3fed60 in term 0.
I20260924 13:42:43.572860 26887 leader_election.cc:302] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e51301c17dd4526937621aeba3fed60, aded32df8cd54b0d9b2861430781094e; no voters:
I20260924 13:42:43.572980 27036 raft_consensus.cc:2781] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:43.573041 27036 raft_consensus.cc:482] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:43.573058 27036 raft_consensus.cc:3037] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:43.573757 27036 raft_consensus.cc:504] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } } peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } }
I20260924 13:42:43.573894 27036 leader_election.cc:288] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [CANDIDATE]: Term 1 election: Requested vote from peers aded32df8cd54b0d9b2861430781094e (127.19.53.193:39725), 8de5b4cde81d40c3bcee0982b73f5573 (127.19.53.194:37865)
I20260924 13:42:43.574086 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508b215ff77a4bda9ac6e06f6c827918" candidate_uuid: "3e51301c17dd4526937621aeba3fed60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5b4cde81d40c3bcee0982b73f5573"
I20260924 13:42:43.574115 26678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508b215ff77a4bda9ac6e06f6c827918" candidate_uuid: "3e51301c17dd4526937621aeba3fed60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aded32df8cd54b0d9b2861430781094e"
I20260924 13:42:43.574182 26678 raft_consensus.cc:3037] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:43.574184 26816 raft_consensus.cc:3037] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:43.574792 26678 raft_consensus.cc:2454] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e51301c17dd4526937621aeba3fed60 in term 1.
I20260924 13:42:43.574949 26816 raft_consensus.cc:2454] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e51301c17dd4526937621aeba3fed60 in term 1.
I20260924 13:42:43.574990 26887 leader_election.cc:302] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e51301c17dd4526937621aeba3fed60, aded32df8cd54b0d9b2861430781094e; no voters:
I20260924 13:42:43.575088 27036 raft_consensus.cc:2781] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:43.575227 27036 raft_consensus.cc:686] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [term 1 LEADER]: Becoming Leader. State: Replica: 3e51301c17dd4526937621aeba3fed60, State: Running, Role: LEADER
I20260924 13:42:43.575328 27036 consensus_queue.cc:237] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } } peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } }
I20260924 13:42:43.576539 25760 catalog_manager.cc:5495] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e51301c17dd4526937621aeba3fed60 (127.19.53.195). New cstate: current_term: 1 leader_uuid: "3e51301c17dd4526937621aeba3fed60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } health_report { overall_health: UNKNOWN } } }
W20260924 13:42:43.602722 27003 tablet.cc:2369] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:42:43.609264 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.606981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.19.53.195:35269)
0924 13:42:43.607057 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.607059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.607068 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:43.607106 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.607107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.607123 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.607183 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:43.607298 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.607926 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.608646 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.608828 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.608881 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.609005 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:43.609113 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:43.609178 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.609194 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.609210 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 13:42:43.611102 26816 raft_consensus.cc:1261] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573 [term 1 FOLLOWER]: Refusing update from remote peer 3e51301c17dd4526937621aeba3fed60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:43.611136 26678 raft_consensus.cc:1261] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e [term 1 FOLLOWER]: Refusing update from remote peer 3e51301c17dd4526937621aeba3fed60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:43.611478 27039 consensus_queue.cc:1035] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:43.611617 27036 consensus_queue.cc:1035] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:43.614606 27044 mvcc.cc:205] Tried to move back new op lower bound from 7332894161348435968 to 7332894161205383168. Current Snapshot: MvccSnapshot[applied={T|T < 7332894161348435968}]
I20260924 13:42:43.616623 27043 mvcc.cc:205] Tried to move back new op lower bound from 7332894161348435968 to 7332894161205383168. Current Snapshot: MvccSnapshot[applied={T|T < 7332894161348435968}]
I20260924 13:42:43.617324 25761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:43.617419 25761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:43.619127 25761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ee85e1707af4b85bd01c90c229805a2: Sending DeleteTablet for 3 replicas of tablet 508b215ff77a4bda9ac6e06f6c827918
I20260924 13:42:43.619428 26658 tablet_service.cc:1514] Processing DeleteTablet for tablet 508b215ff77a4bda9ac6e06f6c827918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:43 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:54442
I20260924 13:42:43.619453 26934 tablet_service.cc:1514] Processing DeleteTablet for tablet 508b215ff77a4bda9ac6e06f6c827918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:43 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:59884
I20260924 13:42:43.619477 26796 tablet_service.cc:1514] Processing DeleteTablet for tablet 508b215ff77a4bda9ac6e06f6c827918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:43 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:33374
I20260924 13:42:43.619616 27052 tablet_replica.cc:330] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e: stopping tablet replica
I20260924 13:42:43.619732 27052 raft_consensus.cc:2229] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:43.619786 27054 tablet_replica.cc:330] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573: stopping tablet replica
I20260924 13:42:43.619844 27052 raft_consensus.cc:2258] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:43.620013 27054 raft_consensus.cc:2229] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:43.620124 27054 raft_consensus.cc:2258] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:43.620296 27052 ts_tablet_manager.cc:1905] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:43.620571 27054 ts_tablet_manager.cc:1905] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:43.621106 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 25729
I20260924 13:42:43.621943 27052 ts_tablet_manager.cc:1918] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:43.622010 27052 log.cc:1196] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-0/wal/wals/508b215ff77a4bda9ac6e06f6c827918
I20260924 13:42:43.622293 27052 ts_tablet_manager.cc:1939] T 508b215ff77a4bda9ac6e06f6c827918 P aded32df8cd54b0d9b2861430781094e: Deleting consensus metadata
I20260924 13:42:43.622905 27054 ts_tablet_manager.cc:1918] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:43.622964 27054 log.cc:1196] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/508b215ff77a4bda9ac6e06f6c827918
I20260924 13:42:43.623230 27054 ts_tablet_manager.cc:1939] T 508b215ff77a4bda9ac6e06f6c827918 P 8de5b4cde81d40c3bcee0982b73f5573: Deleting consensus metadata
W20260924 13:42:43.627902 26751 connection.cc:535] server connection from 127.0.0.1:33374 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 13:42:43.628130 26612 connection.cc:535] server connection from 127.0.0.1:54442 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 13:42:43.630221 27053 tablet_replica.cc:330] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60: stopping tablet replica
I20260924 13:42:43.630326 27053 raft_consensus.cc:2229] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:42:43.630440 27053 raft_consensus.cc:2258] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:43.630875 27053 ts_tablet_manager.cc:1905] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:43.632455 27053 ts_tablet_manager.cc:1918] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:43.632520 27053 log.cc:1196] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/508b215ff77a4bda9ac6e06f6c827918
I20260924 13:42:43.632771 27053 ts_tablet_manager.cc:1939] T 508b215ff77a4bda9ac6e06f6c827918 P 3e51301c17dd4526937621aeba3fed60: Deleting consensus metadata
I20260924 13:42:43.636497 27012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.620698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.19.53.252:36097)
0924 13:42:43.620740 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.620743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.626149 (+ 5406us) server_negotiation.cc:407] Connection header received
0924 13:42:43.626213 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.626214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.626228 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.626283 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.627189 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.627986 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.629371 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.629685 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.629725 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.632070 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.632071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.632072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.632080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.634579 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.635290 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.635291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.635291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.635379 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.635692 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.635693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.635694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.635854 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.635968 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.636219 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.636369 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:42:43.637480 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.629958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.19.53.253:41489)
0924 13:42:43.630007 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.630009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.630018 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:43.630039 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.630040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.630055 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.630109 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.630984 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.631676 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.632756 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.633089 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.633147 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.634516 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.634518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.634518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.634525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.636214 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.636653 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.636654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.636655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.636719 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.636917 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.636918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.636918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.637054 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.637119 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.637311 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.637389 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:42:43.638346 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:43.630134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.19.53.252:36097)
0924 13:42:43.630170 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:43.630172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:43.630395 (+ 223us) server_negotiation.cc:407] Connection header received
0924 13:42:43.630496 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:43.630497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:43.630508 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:43.630537 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:43.631238 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.632008 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.633043 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:43.633353 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:43.633395 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:43.635021 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:43.635022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:43.635023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:43.635029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:43.636930 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.637446 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.637447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.637447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.637498 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:43.637725 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:43.637725 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:43.637726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:43.637855 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:43.637982 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:43.638184 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 13:42:43.638260 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 13:42:43.642210 26886 connection.cc:441] server connection from 127.0.0.1:59884 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59884 (request call id 1) could send its response
I20260924 13:42:44.616040 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:44.615733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.194:51683)
0924 13:42:44.615872 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:44.615965 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
W20260924 13:42:44.616266 26864 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:34111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
I20260924 13:42:44.616991 27063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:44.616781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.195:50977)
0924 13:42:44.616886 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:44.616955 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
W20260924 13:42:44.617283 27001 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:34111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
I20260924 13:42:44.617733 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:44.617498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.193:33179)
0924 13:42:44.617606 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:44.617693 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":48,"threads_started":1}
W20260924 13:42:44.618067 26725 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:34111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
I20260924 13:42:45.119307 27065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8ee85e1707af4b85bd01c90c229805a2)
I20260924 13:42:45.119306 27066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8ee85e1707af4b85bd01c90c229805a2)
I20260924 13:42:45.119483 27066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } }
I20260924 13:42:45.119504 27065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } }
I20260924 13:42:45.119776 27066 leader_election.cc:288] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ee85e1707af4b85bd01c90c229805a2 (127.19.53.254:34111), 7e5fa5965fd84e7f964d462f6ae37efe (127.19.53.253:41489)
I20260924 13:42:45.119782 27065 leader_election.cc:288] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ee85e1707af4b85bd01c90c229805a2 (127.19.53.254:34111), 68acfc076b894170aa626af998518c5a (127.19.53.252:36097)
I20260924 13:42:45.120108 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.119876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:46126)
0924 13:42:45.120005 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:45.120059 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
W20260924 13:42:45.120262 26344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
I20260924 13:42:45.120296 27070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.119878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:46130)
0924 13:42:45.120213 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:45.120257 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":54,"threads_started":1}
W20260924 13:42:45.120406 26531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
I20260924 13:42:45.120826 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.120742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:46138)
0924 13:42:45.120776 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:45.120799 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 13:42:45.120826 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.120742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:46136)
0924 13:42:45.120777 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:45.120800 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 13:42:45.120930 26531 leader_election.cc:334] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8ee85e1707af4b85bd01c90c229805a2 (127.19.53.254:34111): Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
W20260924 13:42:45.121097 26344 leader_election.cc:334] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8ee85e1707af4b85bd01c90c229805a2 (127.19.53.254:34111): Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
I20260924 13:42:45.123090 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.120089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.19.53.252:36097)
0924 13:42:45.120167 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:45.120170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:45.120179 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:45.120208 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:45.120209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:45.120221 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:45.120541 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:45.120583 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:45.121406 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:45.122405 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:45.122755 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:45.122842 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:45.122852 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:45.122994 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 13:42:45.123033 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":61,"thread_start_us":37,"threads_started":1}
I20260924 13:42:45.123090 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.119969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:36097 (local address 127.0.0.1:56930)
0924 13:42:45.120044 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:45.120054 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:45.120105 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:45.120326 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:45.120328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:45.120343 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:45.120411 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:45.120413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:45.121491 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:45.121492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:45.122308 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:45.122310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:45.122844 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:45.122845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:45.122908 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:45.122911 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:45.122986 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 13:42:45.123026 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"thread_start_us":38,"threads_started":1}
I20260924 13:42:45.123312 27072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.120158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.19.53.253:41489)
0924 13:42:45.120331 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:45.120595 (+ 264us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:45.120603 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:45.120624 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:45.120625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:45.120638 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:45.120815 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:45.120845 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:45.121561 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:45.122527 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:45.122874 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:45.122968 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:45.122978 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:45.123116 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 13:42:45.123222 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":49,"threads_started":1}
I20260924 13:42:45.123231 27070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.120046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:41489 (local address 127.0.0.1:48816)
0924 13:42:45.120355 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:45.120363 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:45.120417 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:45.120727 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:45.120728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:45.120731 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:45.120760 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:45.120761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:45.121638 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:45.121639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:45.122425 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:45.122428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:45.122972 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:45.122973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:45.123035 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:45.123038 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:45.123109 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 13:42:45.123146 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":35,"threads_started":1}
I20260924 13:42:45.123636 26552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "68acfc076b894170aa626af998518c5a" is_pre_election: true
I20260924 13:42:45.123739 26552 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e5fa5965fd84e7f964d462f6ae37efe in term 1.
I20260924 13:42:45.123756 26368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68acfc076b894170aa626af998518c5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" is_pre_election: true
I20260924 13:42:45.123858 26368 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68acfc076b894170aa626af998518c5a in term 1.
I20260924 13:42:45.123881 26345 leader_election.cc:302] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68acfc076b894170aa626af998518c5a, 7e5fa5965fd84e7f964d462f6ae37efe; no voters: 8ee85e1707af4b85bd01c90c229805a2
I20260924 13:42:45.123978 27065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 13:42:45.124013 27065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8ee85e1707af4b85bd01c90c229805a2)
I20260924 13:42:45.124038 27065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:42:45.124063 26531 leader_election.cc:302] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68acfc076b894170aa626af998518c5a, 7e5fa5965fd84e7f964d462f6ae37efe; no voters: 8ee85e1707af4b85bd01c90c229805a2
I20260924 13:42:45.124164 27066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 13:42:45.124212 27066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8ee85e1707af4b85bd01c90c229805a2)
I20260924 13:42:45.124228 27066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:42:45.124815 27065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } }
I20260924 13:42:45.124961 27065 leader_election.cc:288] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [CANDIDATE]: Term 2 election: Requested vote from peers 8ee85e1707af4b85bd01c90c229805a2 (127.19.53.254:34111), 68acfc076b894170aa626af998518c5a (127.19.53.252:36097)
I20260924 13:42:45.124912 27066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } }
I20260924 13:42:45.125061 27066 leader_election.cc:288] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [CANDIDATE]: Term 2 election: Requested vote from peers 8ee85e1707af4b85bd01c90c229805a2 (127.19.53.254:34111), 7e5fa5965fd84e7f964d462f6ae37efe (127.19.53.253:41489)
I20260924 13:42:45.125115 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.125031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:46140)
0924 13:42:45.125068 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:45.125091 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 13:42:45.125123 26552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "68acfc076b894170aa626af998518c5a"
I20260924 13:42:45.125214 26552 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e5fa5965fd84e7f964d462f6ae37efe in current term 2: Already voted for candidate 68acfc076b894170aa626af998518c5a in this term.
I20260924 13:42:45.125344 27070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.125218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:46144)
0924 13:42:45.125291 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:45.125319 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 13:42:45.125344 26368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68acfc076b894170aa626af998518c5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7e5fa5965fd84e7f964d462f6ae37efe"
I20260924 13:42:45.125412 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.125335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:46148)
0924 13:42:45.125370 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:45.125391 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 13:42:45.125451 26368 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 68acfc076b894170aa626af998518c5a in current term 2: Already voted for candidate 7e5fa5965fd84e7f964d462f6ae37efe in this term.
W20260924 13:42:45.125458 26344 leader_election.cc:334] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8ee85e1707af4b85bd01c90c229805a2 (127.19.53.254:34111): Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
I20260924 13:42:45.125552 26344 leader_election.cc:302] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e5fa5965fd84e7f964d462f6ae37efe; no voters: 68acfc076b894170aa626af998518c5a, 8ee85e1707af4b85bd01c90c229805a2
I20260924 13:42:45.125641 27070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.125543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:46164)
0924 13:42:45.125596 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:45.125621 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 13:42:45.125717 27065 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260924 13:42:45.125761 26531 leader_election.cc:334] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8ee85e1707af4b85bd01c90c229805a2 (127.19.53.254:34111): Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
I20260924 13:42:45.125799 26531 leader_election.cc:302] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68acfc076b894170aa626af998518c5a; no voters: 7e5fa5965fd84e7f964d462f6ae37efe, 8ee85e1707af4b85bd01c90c229805a2
I20260924 13:42:45.125849 27066 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 13:42:45.616860 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.616581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.194:50515)
0924 13:42:45.616752 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:45.616795 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260924 13:42:45.617772 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.617592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.195:53077)
0924 13:42:45.617697 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:45.617728 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
I20260924 13:42:45.618594 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:45.618393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.193:38793)
0924 13:42:45.618511 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:45.618551 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260924 13:42:46.617617 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.617320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.194:55835)
0924 13:42:46.617507 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.617558 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
I20260924 13:42:46.618352 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.618182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.195:37105)
0924 13:42:46.618283 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.618317 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
I20260924 13:42:46.619215 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.619020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.193:54637)
0924 13:42:46.619133 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.619172 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":47,"threads_started":1}
I20260924 13:42:46.826802 27081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:46.826900 27081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } }
I20260924 13:42:46.827205 27081 leader_election.cc:288] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8ee85e1707af4b85bd01c90c229805a2 (127.19.53.254:34111), 68acfc076b894170aa626af998518c5a (127.19.53.252:36097)
I20260924 13:42:46.827546 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.827343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:49418)
0924 13:42:46.827469 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.827512 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260924 13:42:46.827505 26552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "68acfc076b894170aa626af998518c5a" is_pre_election: true
I20260924 13:42:46.827669 26552 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e5fa5965fd84e7f964d462f6ae37efe in term 2.
I20260924 13:42:46.827862 26345 leader_election.cc:302] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68acfc076b894170aa626af998518c5a, 7e5fa5965fd84e7f964d462f6ae37efe; no voters:
I20260924 13:42:46.827982 27081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 13:42:46.828024 27081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:46.828054 27081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 2 FOLLOWER]: Advancing to term 3
I20260924 13:42:46.828099 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.827927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:49422)
0924 13:42:46.827997 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.828036 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 13:42:46.828233 26344 leader_election.cc:334] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8ee85e1707af4b85bd01c90c229805a2 (127.19.53.254:34111): Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
I20260924 13:42:46.828883 27081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } }
I20260924 13:42:46.829028 27081 leader_election.cc:288] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [CANDIDATE]: Term 3 election: Requested vote from peers 8ee85e1707af4b85bd01c90c229805a2 (127.19.53.254:34111), 68acfc076b894170aa626af998518c5a (127.19.53.252:36097)
I20260924 13:42:46.829254 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.829133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:49438)
0924 13:42:46.829190 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.829225 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 13:42:46.829233 26552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "68acfc076b894170aa626af998518c5a"
I20260924 13:42:46.829314 26552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 2 FOLLOWER]: Advancing to term 3
I20260924 13:42:46.829596 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.829513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:49444)
0924 13:42:46.829547 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.829575 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 13:42:46.829681 26344 leader_election.cc:334] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8ee85e1707af4b85bd01c90c229805a2 (127.19.53.254:34111): Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
I20260924 13:42:46.830195 26552 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5fa5965fd84e7f964d462f6ae37efe in term 3.
I20260924 13:42:46.830371 26345 leader_election.cc:302] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68acfc076b894170aa626af998518c5a, 7e5fa5965fd84e7f964d462f6ae37efe; no voters: 8ee85e1707af4b85bd01c90c229805a2
I20260924 13:42:46.830493 27081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 3 FOLLOWER]: Leader election won for term 3
I20260924 13:42:46.830615 27081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [term 3 LEADER]: Becoming Leader. State: Replica: 7e5fa5965fd84e7f964d462f6ae37efe, State: Running, Role: LEADER
I20260924 13:42:46.830706 27081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } }
I20260924 13:42:46.831053 27084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e5fa5965fd84e7f964d462f6ae37efe. Latest consensus state: current_term: 3 leader_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } } }
I20260924 13:42:46.831153 27084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:46.831322 27086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:46.831681 27086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:46.831734 27086 catalog_manager.cc:1259] Loaded cluster ID: 2e5a03513b4546cd9964bbd188ba433d
I20260924 13:42:46.831750 27086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:46.831884 27086 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:46.831934 27086 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: Loaded TSK: 0
I20260924 13:42:46.832034 27086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:42:46.884654 26357 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:46.886461 26357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:46.887516 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.887383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:49450)
0924 13:42:46.887433 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.887485 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 13:42:46.887563 26552 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [term 3 FOLLOWER]: Refusing update from remote peer 7e5fa5965fd84e7f964d462f6ae37efe: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 13:42:46.887846 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.887759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.0.0.1:49462)
0924 13:42:46.887796 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.887818 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 13:42:46.887796 27084 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [LEADER]: Connected to new peer: Peer: permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 13:42:46.887928 26344 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe -> Peer 8ee85e1707af4b85bd01c90c229805a2 (127.19.53.254:34111): Couldn't send request to peer 8ee85e1707af4b85bd01c90c229805a2. Status: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 13:42:46.888500 27088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e5fa5965fd84e7f964d462f6ae37efe. Latest consensus state: current_term: 3 leader_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } } }
I20260924 13:42:46.888615 27088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:46.889083 27089 mvcc.cc:205] Tried to move back new op lower bound from 7332894174769311744 to 7332894174539296768. Current Snapshot: MvccSnapshot[applied={T|T < 7332894174769311744}]
I20260924 13:42:46.889118 27081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } } }
I20260924 13:42:46.889216 27081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:46.889458 27088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8ee85e1707af4b85bd01c90c229805a2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 34111 } } peers { permanent_uuid: "7e5fa5965fd84e7f964d462f6ae37efe" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 41489 } attrs { promote: false } } peers { permanent_uuid: "68acfc076b894170aa626af998518c5a" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 36097 } attrs { promote: false } } }
I20260924 13:42:46.889588 27088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68acfc076b894170aa626af998518c5a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:46.894042 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.891107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:35269 (local address 127.0.0.1:39846)
0924 13:42:46.891375 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.891386 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:46.891469 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:46.891678 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:46.891679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:46.891681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:46.891729 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:46.891730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.892433 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.892434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:46.893120 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:46.893122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.893780 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.893781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:46.893837 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:46.893841 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:46.893894 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 13:42:46.893970 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":18,"thread_start_us":136,"threads_started":1}
I20260924 13:42:46.894066 27095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.891201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.19.53.195:35269)
0924 13:42:46.891322 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:46.891324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:46.891443 (+ 119us) server_negotiation.cc:407] Connection header received
0924 13:42:46.891540 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:46.891542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:46.891554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:46.891627 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:46.891803 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.892360 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.893210 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.893674 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.893739 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:46.893751 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:46.893902 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 13:42:46.893995 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
I20260924 13:42:46.894536 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.891105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:37865 (local address 127.0.0.1:59290)
0924 13:42:46.891153 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.891180 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:46.891244 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:46.891585 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:46.891586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:46.891587 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:46.891638 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:46.891639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.892354 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.892355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:46.893513 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:46.893515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.894306 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.894307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:46.894371 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:46.894376 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:42:46.894448 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 13:42:46.894486 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 13:42:46.894580 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.891330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.19.53.194:37865)
0924 13:42:46.891427 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:46.891429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:46.891437 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:46.891477 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:46.891478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:46.891489 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:46.891582 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:46.891714 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.892278 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.893592 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.894214 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.894307 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:46.894319 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:46.894465 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 13:42:46.894513 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":49,"threads_started":1}
I20260924 13:42:46.894809 26796 tablet_service.cc:1467] Processing CreateTablet for tablet 37609c9503da4a2dadcb536c0d2a8057 (DEFAULT_TABLE table=test-table [id=2461488ea73b431990b3aa88f077bfc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:46.894908 26934 tablet_service.cc:1467] Processing CreateTablet for tablet 37609c9503da4a2dadcb536c0d2a8057 (DEFAULT_TABLE table=test-table [id=2461488ea73b431990b3aa88f077bfc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:46.894938 26796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37609c9503da4a2dadcb536c0d2a8057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:46.895015 26934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37609c9503da4a2dadcb536c0d2a8057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:46.895653 27097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.891372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39725 (local address 127.0.0.1:49522)
0924 13:42:46.891552 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.891561 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:46.891671 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:46.892643 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:46.892644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:46.892645 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:46.892679 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:46.892680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.893539 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.893540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:46.894646 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:46.894649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.895374 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.895375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:46.895456 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:46.895460 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:46.895534 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 13:42:46.895566 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":1,"thread_start_us":61,"threads_started":1}
I20260924 13:42:46.895689 27098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.891449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.19.53.193:39725)
0924 13:42:46.892489 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:46.892491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:46.892499 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:46.892523 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:46.892524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:46.892534 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:46.892602 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:46.892746 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.893459 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.894746 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.895277 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.895355 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:46.895367 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:46.895556 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 13:42:46.895609 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1020,"thread_start_us":50,"threads_started":1}
I20260924 13:42:46.896346 27099 tablet_bootstrap.cc:492] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573: Bootstrap starting.
I20260924 13:42:46.896435 27100 tablet_bootstrap.cc:492] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60: Bootstrap starting.
I20260924 13:42:46.896692 26658 tablet_service.cc:1467] Processing CreateTablet for tablet 37609c9503da4a2dadcb536c0d2a8057 (DEFAULT_TABLE table=test-table [id=2461488ea73b431990b3aa88f077bfc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:46.896747 27099 tablet_bootstrap.cc:654] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:46.896863 26658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37609c9503da4a2dadcb536c0d2a8057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:46.896934 27100 tablet_bootstrap.cc:654] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:46.897452 27099 tablet_bootstrap.cc:492] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573: No bootstrap required, opened a new log
I20260924 13:42:46.897486 27099 ts_tablet_manager.cc:1397] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:46.897583 27099 raft_consensus.cc:348] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } } peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } }
I20260924 13:42:46.897625 27099 raft_consensus.cc:374] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:46.897639 27099 raft_consensus.cc:729] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de5b4cde81d40c3bcee0982b73f5573, State: Initialized, Role: FOLLOWER
I20260924 13:42:46.897672 27099 consensus_queue.cc:260] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } } peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } }
I20260924 13:42:46.897817 27099 ts_tablet_manager.cc:1428] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:46.898262 27104 tablet_bootstrap.cc:492] T 37609c9503da4a2dadcb536c0d2a8057 P aded32df8cd54b0d9b2861430781094e: Bootstrap starting.
I20260924 13:42:46.898294 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.898001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.194:52929)
0924 13:42:46.898236 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.898267 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
I20260924 13:42:46.898624 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.898525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.194:58309)
0924 13:42:46.898563 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.898595 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 13:42:46.898631 26865 heartbeater.cc:503] Master 127.19.53.253:41489 was elected leader, sending a full tablet report...
I20260924 13:42:46.898689 27104 tablet_bootstrap.cc:654] T 37609c9503da4a2dadcb536c0d2a8057 P aded32df8cd54b0d9b2861430781094e: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:46.899240 27100 tablet_bootstrap.cc:492] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60: No bootstrap required, opened a new log
I20260924 13:42:46.899307 27100 ts_tablet_manager.cc:1397] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 13:42:46.899434 27100 raft_consensus.cc:348] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } } peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } }
I20260924 13:42:46.899510 27100 raft_consensus.cc:374] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:46.899541 27100 raft_consensus.cc:729] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e51301c17dd4526937621aeba3fed60, State: Initialized, Role: FOLLOWER
I20260924 13:42:46.899591 27100 consensus_queue.cc:260] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } } peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } }
I20260924 13:42:46.899746 27100 ts_tablet_manager.cc:1428] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:46.900027 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.899894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.195:44589)
0924 13:42:46.899969 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.900000 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 13:42:46.900331 27002 heartbeater.cc:503] Master 127.19.53.253:41489 was elected leader, sending a full tablet report...
I20260924 13:42:46.900334 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.900253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.195:33631)
0924 13:42:46.900286 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.900311 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 13:42:46.900579 27104 tablet_bootstrap.cc:492] T 37609c9503da4a2dadcb536c0d2a8057 P aded32df8cd54b0d9b2861430781094e: No bootstrap required, opened a new log
I20260924 13:42:46.900656 27104 ts_tablet_manager.cc:1397] T 37609c9503da4a2dadcb536c0d2a8057 P aded32df8cd54b0d9b2861430781094e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:46.900816 27104 raft_consensus.cc:348] T 37609c9503da4a2dadcb536c0d2a8057 P aded32df8cd54b0d9b2861430781094e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } } peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } }
I20260924 13:42:46.900887 27104 raft_consensus.cc:374] T 37609c9503da4a2dadcb536c0d2a8057 P aded32df8cd54b0d9b2861430781094e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:46.900919 27104 raft_consensus.cc:729] T 37609c9503da4a2dadcb536c0d2a8057 P aded32df8cd54b0d9b2861430781094e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aded32df8cd54b0d9b2861430781094e, State: Initialized, Role: FOLLOWER
I20260924 13:42:46.900980 27104 consensus_queue.cc:260] T 37609c9503da4a2dadcb536c0d2a8057 P aded32df8cd54b0d9b2861430781094e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } } peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } }
I20260924 13:42:46.901146 27104 ts_tablet_manager.cc:1428] T 37609c9503da4a2dadcb536c0d2a8057 P aded32df8cd54b0d9b2861430781094e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:46.901417 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.901326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.193:39865)
0924 13:42:46.901359 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.901389 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 13:42:46.901711 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.901634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.193:44977)
0924 13:42:46.901662 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.901688 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260924 13:42:46.974004 27106 raft_consensus.cc:482] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:46.974123 27106 raft_consensus.cc:504] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } } peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } }
I20260924 13:42:46.974339 27106 leader_election.cc:288] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8de5b4cde81d40c3bcee0982b73f5573 (127.19.53.194:37865), aded32df8cd54b0d9b2861430781094e (127.19.53.193:39725)
I20260924 13:42:46.974545 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37609c9503da4a2dadcb536c0d2a8057" candidate_uuid: "3e51301c17dd4526937621aeba3fed60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5b4cde81d40c3bcee0982b73f5573" is_pre_election: true
I20260924 13:42:46.974552 26678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37609c9503da4a2dadcb536c0d2a8057" candidate_uuid: "3e51301c17dd4526937621aeba3fed60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aded32df8cd54b0d9b2861430781094e" is_pre_election: true
I20260924 13:42:46.974637 26816 raft_consensus.cc:2454] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e51301c17dd4526937621aeba3fed60 in term 0.
I20260924 13:42:46.974658 26678 raft_consensus.cc:2454] T 37609c9503da4a2dadcb536c0d2a8057 P aded32df8cd54b0d9b2861430781094e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e51301c17dd4526937621aeba3fed60 in term 0.
I20260924 13:42:46.974797 26889 leader_election.cc:302] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e51301c17dd4526937621aeba3fed60, 8de5b4cde81d40c3bcee0982b73f5573; no voters:
I20260924 13:42:46.974910 27106 raft_consensus.cc:2781] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:46.974951 27106 raft_consensus.cc:482] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:46.974982 27106 raft_consensus.cc:3037] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:46.975664 27106 raft_consensus.cc:504] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } } peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } }
I20260924 13:42:46.975806 27106 leader_election.cc:288] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [CANDIDATE]: Term 1 election: Requested vote from peers 8de5b4cde81d40c3bcee0982b73f5573 (127.19.53.194:37865), aded32df8cd54b0d9b2861430781094e (127.19.53.193:39725)
I20260924 13:42:46.975960 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37609c9503da4a2dadcb536c0d2a8057" candidate_uuid: "3e51301c17dd4526937621aeba3fed60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5b4cde81d40c3bcee0982b73f5573"
I20260924 13:42:46.975980 26678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37609c9503da4a2dadcb536c0d2a8057" candidate_uuid: "3e51301c17dd4526937621aeba3fed60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aded32df8cd54b0d9b2861430781094e"
I20260924 13:42:46.976029 26816 raft_consensus.cc:3037] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:46.976042 26678 raft_consensus.cc:3037] T 37609c9503da4a2dadcb536c0d2a8057 P aded32df8cd54b0d9b2861430781094e [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:46.976761 26816 raft_consensus.cc:2454] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e51301c17dd4526937621aeba3fed60 in term 1.
I20260924 13:42:46.976761 26678 raft_consensus.cc:2454] T 37609c9503da4a2dadcb536c0d2a8057 P aded32df8cd54b0d9b2861430781094e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e51301c17dd4526937621aeba3fed60 in term 1.
I20260924 13:42:46.976927 26887 leader_election.cc:302] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e51301c17dd4526937621aeba3fed60, aded32df8cd54b0d9b2861430781094e; no voters:
I20260924 13:42:46.977041 27106 raft_consensus.cc:2781] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:46.977185 27106 raft_consensus.cc:686] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [term 1 LEADER]: Becoming Leader. State: Replica: 3e51301c17dd4526937621aeba3fed60, State: Running, Role: LEADER
I20260924 13:42:46.977231 27106 consensus_queue.cc:237] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } } peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } }
I20260924 13:42:46.977506 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.977409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.195:40965)
0924 13:42:46.977445 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.977476 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 13:42:46.977878 26358 catalog_manager.cc:5495] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e51301c17dd4526937621aeba3fed60 (127.19.53.195). New cstate: current_term: 1 leader_uuid: "3e51301c17dd4526937621aeba3fed60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e51301c17dd4526937621aeba3fed60" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 } health_report { overall_health: UNKNOWN } } }
I20260924 13:42:46.986135 27095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.983979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.19.53.195:35269)
0924 13:42:46.984020 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:46.984022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:46.984031 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:46.984071 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:46.984072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:46.984098 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:46.984151 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:46.984248 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.984754 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.985490 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:46.985668 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:46.985718 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:46.985827 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:46.985851 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:46.985886 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:46.986008 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 13:42:46.986028 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:46.987075 26678 raft_consensus.cc:1261] T 37609c9503da4a2dadcb536c0d2a8057 P aded32df8cd54b0d9b2861430781094e [term 1 FOLLOWER]: Refusing update from remote peer 3e51301c17dd4526937621aeba3fed60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:46.987224 26816 raft_consensus.cc:1261] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573 [term 1 FOLLOWER]: Refusing update from remote peer 3e51301c17dd4526937621aeba3fed60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:46.987277 27106 consensus_queue.cc:1035] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aded32df8cd54b0d9b2861430781094e" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:46.987527 27106 consensus_queue.cc:1035] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5b4cde81d40c3bcee0982b73f5573" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:46.987704 26726 heartbeater.cc:503] Master 127.19.53.253:41489 was elected leader, sending a full tablet report...
I20260924 13:42:46.987972 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.987849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.193:38635)
0924 13:42:46.987900 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.987944 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 13:42:46.988678 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.988581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.194:55819)
0924 13:42:46.988624 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.988649 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 13:42:46.989346 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.989259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.195:37715)
0924 13:42:46.989290 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.989320 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 13:42:46.989795 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.989710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.195:54549)
0924 13:42:46.989744 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.989766 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 13:42:46.990434 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.990259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.193:57523)
0924 13:42:46.990385 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.990409 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 13:42:46.990628 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:46.990545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:34111 (local address 127.19.53.194:60629)
0924 13:42:46.990576 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:46.990601 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 13:42:46.991801 26357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:46.991902 26357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:46.993119 26357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e5fa5965fd84e7f964d462f6ae37efe: Sending DeleteTablet for 3 replicas of tablet 37609c9503da4a2dadcb536c0d2a8057
I20260924 13:42:46.993402 26934 tablet_service.cc:1514] Processing DeleteTablet for tablet 37609c9503da4a2dadcb536c0d2a8057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:46 UTC) from {username='oryx', principal='oryx/127.19.53.253@KRBTEST.COM'} at 127.0.0.1:39846
I20260924 13:42:46.993414 26796 tablet_service.cc:1514] Processing DeleteTablet for tablet 37609c9503da4a2dadcb536c0d2a8057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:46 UTC) from {username='oryx', principal='oryx/127.19.53.253@KRBTEST.COM'} at 127.0.0.1:59290
I20260924 13:42:46.993410 26658 tablet_service.cc:1514] Processing DeleteTablet for tablet 37609c9503da4a2dadcb536c0d2a8057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:46 UTC) from {username='oryx', principal='oryx/127.19.53.253@KRBTEST.COM'} at 127.0.0.1:49522
I20260924 13:42:46.993602 27120 tablet_replica.cc:330] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60: stopping tablet replica
I20260924 13:42:46.993659 27120 raft_consensus.cc:2229] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:42:46.993718 27120 raft_consensus.cc:2258] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:46.993786 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 26505
I20260924 13:42:46.994694 27120 ts_tablet_manager.cc:1905] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:46.996578 27120 ts_tablet_manager.cc:1918] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:46.996640 27120 log.cc:1196] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/37609c9503da4a2dadcb536c0d2a8057
I20260924 13:42:46.996871 27120 ts_tablet_manager.cc:1939] T 37609c9503da4a2dadcb536c0d2a8057 P 3e51301c17dd4526937621aeba3fed60: Deleting consensus metadata
I20260924 13:42:46.997201 26343 catalog_manager.cc:4841] TS 3e51301c17dd4526937621aeba3fed60 (127.19.53.195:35269): tablet 37609c9503da4a2dadcb536c0d2a8057 (table test-table [id=2461488ea73b431990b3aa88f077bfc7]) successfully deleted
I20260924 13:42:47.000710 27097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:47.000606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39725 (local address 127.0.0.1:49536)
0924 13:42:47.000657 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:47.000684 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 13:42:47.000880 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 26732
I20260924 13:42:47.000880 27119 tablet_replica.cc:330] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573: stopping tablet replica
I20260924 13:42:47.000955 27119 raft_consensus.cc:2229] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:47.001004 27119 raft_consensus.cc:2258] T 37609c9503da4a2dadcb536c0d2a8057 P 8de5b4cde81d40c3bcee0982b73f5573 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 13:42:47.002738 26343 catalog_manager.cc:4584] TS aded32df8cd54b0d9b2861430781094e (127.19.53.193:39725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37609c9503da4a2dadcb536c0d2a8057: Network error: Client connection negotiation failed: client connection to 127.19.53.193:39725: connect: Connection refused (error 111)
I20260924 13:42:47.005967 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 26870
I20260924 13:42:47.006042 27097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:47.005861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:37865 (local address 127.0.0.1:59296)
0924 13:42:47.005946 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:47.005990 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.194:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 13:42:47.012311 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 26225
I20260924 13:42:47.017094 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 26412
2026-09-24T13:42:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6411 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:47 dist-test-slave-tfm9 krb5kdc[27124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:47 dist-test-slave-tfm9 krb5kdc[27124](info): set up 2 sockets
Sep 24 13:42:47 dist-test-slave-tfm9 krb5kdc[27124](info): commencing operation
krb5kdc: starting...
W20260924 13:42:48.059954 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.254.keytab.
Entry for principal oryx/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.254.keytab.
I20260924 13:42:48.176560 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:40635
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.254.keytab
--principal=oryx/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:40635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:48.255184 27137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:48.255414 27137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:48.255447 27137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:48.256590 27137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:48.256659 27137 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 13:42:48.256680 27137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:48.256702 27137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:48.256731 27137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:48.256757 27137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:48.258229 27137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:40635
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.254.keytab
--principal=oryx/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:40635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:48.258514 27137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:48.258728 27137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:48.263509 27143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:48.263501 27145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:48.263502 27142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:48.263823 27137 server_base.cc:1034] running on GCE node
I20260924 13:42:48.264072 27137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:48.264357 27137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:48.265586 27137 hybrid_clock.cc:648] HybridClock initialized: now 1790257368265572 us; error 34 us; skew 500 ppm
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257368, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:48.268049 27137 init.cc:377] Logged in from keytab as oryx/127.19.53.254@KRBTEST.COM (short username oryx)
I20260924 13:42:48.268527 27137 webserver.cc:460] Webserver started at http://127.19.53.254:40117/ using document root <none> and password file <none>
I20260924 13:42:48.268748 27137 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:48.268814 27137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:48.268957 27137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:48.270056 27137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "936aaeea5ba74a48aebb0587c9329e21"
format_stamp: "Formatted at 2026-09-24 13:42:48 on dist-test-slave-tfm9"
server_key: "cb7850ed014b78f3098e3c3d8f1e573a"
server_key_iv: "04fadc45a52769d9bc3c874abc12dc98"
server_key_version: "encryptionkey@0"
I20260924 13:42:48.270421 27137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "936aaeea5ba74a48aebb0587c9329e21"
format_stamp: "Formatted at 2026-09-24 13:42:48 on dist-test-slave-tfm9"
server_key: "cb7850ed014b78f3098e3c3d8f1e573a"
server_key_iv: "04fadc45a52769d9bc3c874abc12dc98"
server_key_version: "encryptionkey@0"
I20260924 13:42:48.271736 27137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:48.272404 27152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:48.272601 27137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:48.272699 27137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "936aaeea5ba74a48aebb0587c9329e21"
format_stamp: "Formatted at 2026-09-24 13:42:48 on dist-test-slave-tfm9"
server_key: "cb7850ed014b78f3098e3c3d8f1e573a"
server_key_iv: "04fadc45a52769d9bc3c874abc12dc98"
server_key_version: "encryptionkey@0"
I20260924 13:42:48.272768 27137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:48.288877 27137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:48.290748 27137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:48.290915 27137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:48.295440 27137 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:40635
I20260924 13:42:48.295490 27215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:40635 every 8 connection(s)
I20260924 13:42:48.295809 27137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:48.296414 27216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:48.298731 27216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21: Bootstrap starting.
I20260924 13:42:48.299320 27216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:48.299608 27216 log.cc:824] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:48.300328 27216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21: No bootstrap required, opened a new log
I20260924 13:42:48.301060 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 27137
I20260924 13:42:48.301155 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:48.301353 19671 external_mini_cluster.cc:1428] Setting key e1527ac72b6152d923a41617a5347d10
I20260924 13:42:48.302840 27216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "936aaeea5ba74a48aebb0587c9329e21" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 40635 } }
I20260924 13:42:48.302970 27216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:48.303000 27216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 936aaeea5ba74a48aebb0587c9329e21, State: Initialized, Role: FOLLOWER
I20260924 13:42:48.303195 27216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "936aaeea5ba74a48aebb0587c9329e21" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 40635 } }
I20260924 13:42:48.303290 27216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:48.303333 27216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:48.303385 27216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:48.303876 27216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "936aaeea5ba74a48aebb0587c9329e21" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 40635 } }
I20260924 13:42:48.303999 27216 leader_election.cc:302] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 936aaeea5ba74a48aebb0587c9329e21; no voters:
I20260924 13:42:48.304198 27216 leader_election.cc:288] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:48.304255 27221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:48.304445 27221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [term 1 LEADER]: Becoming Leader. State: Replica: 936aaeea5ba74a48aebb0587c9329e21, State: Running, Role: LEADER
I20260924 13:42:48.304508 27216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:48.304582 27221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "936aaeea5ba74a48aebb0587c9329e21" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 40635 } }
I20260924 13:42:48.304970 27221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 936aaeea5ba74a48aebb0587c9329e21. Latest consensus state: current_term: 1 leader_uuid: "936aaeea5ba74a48aebb0587c9329e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "936aaeea5ba74a48aebb0587c9329e21" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 40635 } } }
I20260924 13:42:48.304980 27222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "936aaeea5ba74a48aebb0587c9329e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "936aaeea5ba74a48aebb0587c9329e21" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 40635 } } }
I20260924 13:42:48.306066 27222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:48.306236 27221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21 [sys.catalog]: This master's current role is: LEADER
W20260924 13:42:48.306309 27236 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:48.306489 27236 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:48.306573 27238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:48.306746 27238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:48.309223 27238 catalog_manager.cc:1347] Generated new cluster ID: f9db3ab266424bd59509e9e285166686
I20260924 13:42:48.309322 27238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:48.312975 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.301961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42010 (local address 127.19.53.254:40635)
0924 13:42:48.302168 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.302175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.302199 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:42:48.302824 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.302831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.302878 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.303047 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:48.303790 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.305067 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.306071 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.306678 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.306785 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.309328 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:48.309332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:48.309332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:48.309343 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:48.311495 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.311859 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.311862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.311862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.311965 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.312178 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.312178 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.312179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.312410 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:48.312619 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.312685 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.312818 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260924 13:42:48.322665 27238 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:48.323167 27238 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:48.325152 27238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21: Generated new TSK 0
I20260924 13:42:48.325320 27238 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab.
Entry for principal oryx/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab.
I20260924 13:42:48.354921 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:40635
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab
--principal=oryx/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:48.452453 27242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:48.452656 27242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:48.452679 27242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:48.453851 27242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:48.453893 27242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:48.453970 27242 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:42:48.454010 27242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:48.455510 27242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab
--principal=oryx/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.53.254:40635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:48.455767 27242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:48.455993 27242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:48.461589 27247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:48.461638 27250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:48.461638 27248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:48.461966 27242 server_base.cc:1034] running on GCE node
I20260924 13:42:48.462142 27242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:48.462359 27242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:48.463507 27242 hybrid_clock.cc:648] HybridClock initialized: now 1790257368463492 us; error 34 us; skew 500 ppm
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257368, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:48.466007 27242 init.cc:377] Logged in from keytab as oryx/127.19.53.193@KRBTEST.COM (short username oryx)
I20260924 13:42:48.466449 27242 webserver.cc:460] Webserver started at http://127.19.53.193:35595/ using document root <none> and password file <none>
I20260924 13:42:48.466624 27242 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:48.466660 27242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:48.466744 27242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:48.467715 27242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "817228b4760b4a28b992d85ae1f3fc3a"
format_stamp: "Formatted at 2026-09-24 13:42:48 on dist-test-slave-tfm9"
server_key: "cb47666ceffa36a2d536609cd220ca5a"
server_key_iv: "1485ec1ddb0e899f8719bd42efdf3bc5"
server_key_version: "encryptionkey@0"
I20260924 13:42:48.468003 27242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "817228b4760b4a28b992d85ae1f3fc3a"
format_stamp: "Formatted at 2026-09-24 13:42:48 on dist-test-slave-tfm9"
server_key: "cb47666ceffa36a2d536609cd220ca5a"
server_key_iv: "1485ec1ddb0e899f8719bd42efdf3bc5"
server_key_version: "encryptionkey@0"
I20260924 13:42:48.469296 27242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:48.470090 27257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:48.470249 27242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:48.470324 27242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "817228b4760b4a28b992d85ae1f3fc3a"
format_stamp: "Formatted at 2026-09-24 13:42:48 on dist-test-slave-tfm9"
server_key: "cb47666ceffa36a2d536609cd220ca5a"
server_key_iv: "1485ec1ddb0e899f8719bd42efdf3bc5"
server_key_version: "encryptionkey@0"
I20260924 13:42:48.470377 27242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:48.478178 27242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:48.479890 27242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:48.479997 27242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:48.480397 27242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:48.480432 27242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:48.480454 27242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:48.480466 27242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:48.487263 27242 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:41805
I20260924 13:42:48.487382 27373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:41805 every 8 connection(s)
I20260924 13:42:48.487655 27242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257368, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:48.489984 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 27242
I20260924 13:42:48.490098 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:42:48.490290 19671 external_mini_cluster.cc:1428] Setting key e16d4c46c5d01c88ff1c4ab6f80ae070
I20260924 13:42:48.491371 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.480917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:40635 (local address 127.19.53.193:58029)
0924 13:42:48.481052 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.481085 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.481567 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.481851 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.481857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.482293 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:48.482588 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.482594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.483581 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.483583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.485103 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.485106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.485574 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.485575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.485732 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.486372 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:48.486392 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:48.488185 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:48.489879 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.489887 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.489889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.490173 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.490362 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.490363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.490364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.490528 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.490809 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:48.490814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:48.490994 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.491095 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.491223 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260924 13:42:48.491390 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.480923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:58029 (local address 127.19.53.254:40635)
0924 13:42:48.480966 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.480968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.481166 (+ 198us) server_negotiation.cc:407] Connection header received
0924 13:42:48.481666 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.481668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.481684 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.481748 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:48.482696 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.483480 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.485200 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.485474 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.485530 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.488298 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:48.488300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:48.488300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:48.488307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:48.489763 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.490232 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.490234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.490234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.490296 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.490613 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.490615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.490615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.490745 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:48.490799 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.491169 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.491261 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:48.492014 27374 heartbeater.cc:347] Connected to a master server at 127.19.53.254:40635
I20260924 13:42:48.492102 27374 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:48.492277 27374 heartbeater.cc:511] Master 127.19.53.254:40635 requested a full tablet report, sending...
I20260924 13:42:48.492784 27168 ts_manager.cc:194] Registered new tserver with Master: 817228b4760b4a28b992d85ae1f3fc3a (127.19.53.193:41805)
I20260924 13:42:48.493427 27168 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:58029
WARNING: no policy specified for oryx/127.19.53.194@KRBTEST.COM; defaulting to no policy
I20260924 13:42:48.499398 27381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.493740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:40635 (local address 127.19.53.193:45527)
0924 13:42:48.493862 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.493873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.493965 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.494102 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.494103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.494331 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:48.494396 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.494398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.495014 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.495015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.495534 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.495535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.495898 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.495899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.495970 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.496410 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:48.496415 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:48.496674 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:48.498356 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.498357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.498358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.498551 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.498764 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.498765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.498766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.498823 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.499072 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:48.499073 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:48.499118 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.499205 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.499305 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
I20260924 13:42:48.499394 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.493855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:45527 (local address 127.19.53.254:40635)
0924 13:42:48.493894 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.493896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.493972 (+ 76us) server_negotiation.cc:407] Connection header received
0924 13:42:48.494015 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.494016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.494025 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.494099 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:48.494481 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.494906 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.495623 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.495815 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.495896 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.496794 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:48.496795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:48.496796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:48.496802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:48.498250 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.498636 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.498637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.498638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.498690 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.498894 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.498896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.498896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.499005 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:48.499080 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.499230 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.499305 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab.
Entry for principal oryx/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab.
I20260924 13:42:48.514745 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:40635
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab
--principal=oryx/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:48.595642 27383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:48.595875 27383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:48.595916 27383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:48.597038 27383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:48.597106 27383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:48.597144 27383 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:42:48.597203 27383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:48.598852 27383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab
--principal=oryx/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.53.254:40635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:48.599110 27383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:48.599367 27383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:48.604383 27389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:48.604492 27383 server_base.cc:1034] running on GCE node
W20260924 13:42:48.604422 27388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:48.604616 27391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:48.604847 27383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:48.605122 27383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:48.606388 27383 hybrid_clock.cc:648] HybridClock initialized: now 1790257368606362 us; error 42 us; skew 500 ppm
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257368, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:48.608757 27383 init.cc:377] Logged in from keytab as oryx/127.19.53.194@KRBTEST.COM (short username oryx)
I20260924 13:42:48.609215 27383 webserver.cc:460] Webserver started at http://127.19.53.194:43395/ using document root <none> and password file <none>
I20260924 13:42:48.609443 27383 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:48.609515 27383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:48.609665 27383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:48.610757 27383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "ab4005cdf7604c09a87d7aed0a0be3f1"
format_stamp: "Formatted at 2026-09-24 13:42:48 on dist-test-slave-tfm9"
server_key: "6e84e0d794b3de0832a7a9af663abaa7"
server_key_iv: "874b16a2d67f5834089feea91e97d7a8"
server_key_version: "encryptionkey@0"
I20260924 13:42:48.611133 27383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "ab4005cdf7604c09a87d7aed0a0be3f1"
format_stamp: "Formatted at 2026-09-24 13:42:48 on dist-test-slave-tfm9"
server_key: "6e84e0d794b3de0832a7a9af663abaa7"
server_key_iv: "874b16a2d67f5834089feea91e97d7a8"
server_key_version: "encryptionkey@0"
I20260924 13:42:48.612353 27383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:42:48.613080 27398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:48.613242 27383 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:48.613336 27383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "ab4005cdf7604c09a87d7aed0a0be3f1"
format_stamp: "Formatted at 2026-09-24 13:42:48 on dist-test-slave-tfm9"
server_key: "6e84e0d794b3de0832a7a9af663abaa7"
server_key_iv: "874b16a2d67f5834089feea91e97d7a8"
server_key_version: "encryptionkey@0"
I20260924 13:42:48.613487 27383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:48.628742 27383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:48.630594 27383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:48.630755 27383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:48.631291 27383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:48.631348 27383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:48.631431 27383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:48.631481 27383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:48.638540 27383 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:45089
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257368, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:48.638873 27383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:42:48.639460 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 27383
I20260924 13:42:48.639601 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:42:48.639159 27513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:45089 every 8 connection(s)
I20260924 13:42:48.639853 19671 external_mini_cluster.cc:1428] Setting key 44aecafdbe99f422188d83854c10908d
I20260924 13:42:48.641996 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.631663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:49855 (local address 127.19.53.254:40635)
0924 13:42:48.631711 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.631713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.632024 (+ 311us) server_negotiation.cc:407] Connection header received
0924 13:42:48.632749 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.632751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.632761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.632814 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:48.633687 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.634516 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.636077 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.636372 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.636456 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.639058 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:48.639060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:48.639060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:48.639068 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:48.640631 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.641034 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.641035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.641036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.641098 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.641345 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.641346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.641347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.641466 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:48.641542 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.641801 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.641876 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:42:48.642161 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.631666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:40635 (local address 127.19.53.194:49855)
0924 13:42:48.631839 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.631881 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.632635 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.632892 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.632897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.633298 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:48.633599 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.633605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.634619 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.634621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.635976 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.635979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.636478 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.636479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.636633 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.637123 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:48.637133 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:48.638925 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:48.640725 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.640730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.640732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.640943 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.641173 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.641173 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.641174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.641279 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.641534 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:48.641540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:48.641671 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.641771 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.641914 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260924 13:42:48.642620 27514 heartbeater.cc:347] Connected to a master server at 127.19.53.254:40635
I20260924 13:42:48.642786 27514 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:48.642979 27514 heartbeater.cc:511] Master 127.19.53.254:40635 requested a full tablet report, sending...
I20260924 13:42:48.643404 27168 ts_manager.cc:194] Registered new tserver with Master: ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194:45089)
I20260924 13:42:48.643910 27168 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:49855
WARNING: no policy specified for oryx/127.19.53.195@KRBTEST.COM; defaulting to no policy
I20260924 13:42:48.649408 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.643663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:58597 (local address 127.19.53.254:40635)
0924 13:42:48.643699 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.643702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.643863 (+ 161us) server_negotiation.cc:407] Connection header received
0924 13:42:48.643933 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.643934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.643945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.644000 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:48.644355 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.644895 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.645585 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.645803 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.645876 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.646773 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:48.646775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:48.646775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:48.646781 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:48.648257 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.648603 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.648604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.648605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.648656 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.648857 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.648858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.648859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.649011 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:48.649086 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.649244 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.649316 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 13:42:48.649425 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.643628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:40635 (local address 127.19.53.194:58597)
0924 13:42:48.643795 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.643804 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.643876 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.644005 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.644006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.644198 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:48.644277 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.644279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.644995 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.644996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.645505 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.645506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.645876 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.645877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.645979 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.646404 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:48.646409 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:48.646669 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:48.648347 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.648348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.648348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.648518 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.648729 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.648730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.648730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.648786 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.649090 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:48.649091 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:48.649135 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.649219 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.649318 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":2,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab.
Entry for principal oryx/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab.
I20260924 13:42:48.664773 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:40635
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab
--principal=oryx/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:48.743420 27524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:48.743625 27524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:48.743660 27524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:48.744741 27524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:48.744801 27524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:48.744833 27524 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:42:48.744867 27524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:48.746260 27524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab
--principal=oryx/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.53.254:40635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:48.746493 27524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:48.746721 27524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:48.751606 27532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:48.751611 27529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:48.751842 27524 server_base.cc:1034] running on GCE node
W20260924 13:42:48.751608 27530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:48.752147 27524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:48.752403 27524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:48.753590 27524 hybrid_clock.cc:648] HybridClock initialized: now 1790257368753574 us; error 27 us; skew 500 ppm
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257368, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:48.756062 27524 init.cc:377] Logged in from keytab as oryx/127.19.53.195@KRBTEST.COM (short username oryx)
I20260924 13:42:48.756510 27524 webserver.cc:460] Webserver started at http://127.19.53.195:44157/ using document root <none> and password file <none>
I20260924 13:42:48.756749 27524 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:48.756824 27524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:48.756976 27524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:48.758091 27524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "182cb6f20b2141ce96481e1f6f8658e5"
format_stamp: "Formatted at 2026-09-24 13:42:48 on dist-test-slave-tfm9"
server_key: "972ef1cfe19e5b3cf9143266f0eff24b"
server_key_iv: "f686cd7e28b94a460e8209d8a29ba5d0"
server_key_version: "encryptionkey@0"
I20260924 13:42:48.758471 27524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "182cb6f20b2141ce96481e1f6f8658e5"
format_stamp: "Formatted at 2026-09-24 13:42:48 on dist-test-slave-tfm9"
server_key: "972ef1cfe19e5b3cf9143266f0eff24b"
server_key_iv: "f686cd7e28b94a460e8209d8a29ba5d0"
server_key_version: "encryptionkey@0"
I20260924 13:42:48.759804 27524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:48.760564 27539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:48.760742 27524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:48.760844 27524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "182cb6f20b2141ce96481e1f6f8658e5"
format_stamp: "Formatted at 2026-09-24 13:42:48 on dist-test-slave-tfm9"
server_key: "972ef1cfe19e5b3cf9143266f0eff24b"
server_key_iv: "f686cd7e28b94a460e8209d8a29ba5d0"
server_key_version: "encryptionkey@0"
I20260924 13:42:48.760947 27524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:48.780469 27524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:48.782485 27524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:48.782663 27524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:48.783246 27524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:48.783308 27524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:48.783340 27524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:48.783367 27524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257368, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:48.790722 27524 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:39369
I20260924 13:42:48.790794 27656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:39369 every 8 connection(s)
I20260924 13:42:48.791090 27524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:42:48.793686 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.783665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:46921 (local address 127.19.53.254:40635)
0924 13:42:48.783736 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.783738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.783994 (+ 256us) server_negotiation.cc:407] Connection header received
0924 13:42:48.784733 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.784734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.784747 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.784804 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:48.785711 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.786420 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.788112 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.788354 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.788433 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.791055 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:48.791057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:48.791058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:48.791066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:48.792549 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.792919 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.792920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.792920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.792973 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.793226 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.793227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.793227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.793328 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:48.793379 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.793574 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.793628 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 13:42:48.793845 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.783642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:40635 (local address 127.19.53.195:46921)
0924 13:42:48.783837 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.783891 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.784618 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.784894 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.784899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.785290 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:48.785613 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.785621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.786522 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.786523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.787996 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.787999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.788433 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.788435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.788594 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.789124 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:48.789133 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:48.790963 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:48.792617 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.792622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.792623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.792830 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.793052 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.793053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.793053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.793168 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.793369 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:48.793372 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:48.793478 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.793560 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.793666 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260924 13:42:48.794348 27657 heartbeater.cc:347] Connected to a master server at 127.19.53.254:40635
I20260924 13:42:48.794422 27657 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:48.794620 27657 heartbeater.cc:511] Master 127.19.53.254:40635 requested a full tablet report, sending...
I20260924 13:42:48.794974 27168 ts_manager.cc:194] Registered new tserver with Master: 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195:39369)
I20260924 13:42:48.795321 27168 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:46921
I20260924 13:42:48.799244 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 27524
I20260924 13:42:48.799368 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:42:48.799569 19671 external_mini_cluster.cc:1428] Setting key bd04dbe5cbb47116d33e184cdac5d861
I20260924 13:42:48.800398 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:42:48.800747 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.795411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:40635 (local address 127.19.53.195:34129)
0924 13:42:48.795572 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.795581 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.795638 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.795764 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.795764 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.795945 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:48.796004 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.796005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.796618 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.796619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.797119 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.797121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.797495 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.797496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.797557 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.797893 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:48.797898 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:48.798217 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:48.799677 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.799678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.799679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.799892 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.800097 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.800099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.800099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.800160 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.800435 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:48.800436 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:48.800479 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.800568 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.800654 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":117,"threads_started":1}
I20260924 13:42:48.801084 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.795460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:34129 (local address 127.19.53.254:40635)
0924 13:42:48.795500 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.795502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.795644 (+ 142us) server_negotiation.cc:407] Connection header received
0924 13:42:48.795696 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.795697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.795707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.795759 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:48.796081 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.796530 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.797216 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.797415 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.797495 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.798347 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:48.798348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:48.798349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:48.798356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:48.799550 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.799969 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.799970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.799970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.800024 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.800240 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.800241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.800241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.800350 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:48.800421 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.800599 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.800672 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257368, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 13:42:48.821403 27220 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:48.816112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42012 (local address 127.19.53.254:40635)
0924 13:42:48.816153 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.816156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.816166 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:42:48.816224 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.816225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.816235 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.816299 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:48.816878 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.817408 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.818202 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.818405 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.818496 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.821322 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42012: BlockingRecv error: recv got EOF from 127.0.0.1:42012 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257368, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:48.828964 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.822235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.19.53.254:40635)
0924 13:42:48.822304 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.822306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.822377 (+ 71us) server_negotiation.cc:407] Connection header received
0924 13:42:48.822496 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.822497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.822507 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.822562 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:48.823021 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.823460 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.824154 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.824339 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.824422 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.826691 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:48.826692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:48.826692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:48.826699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:48.827835 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.828188 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.828189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.828190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.828250 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.828458 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.828459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.828459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.828570 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:48.828675 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.828777 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.828854 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 13:42:48.831038 27168 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:48.832574 27168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:48.840153 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.835557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:45089 (local address 127.0.0.1:35076)
0924 13:42:48.836139 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.836149 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.836203 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.836445 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.836446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.836448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:48.836496 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.836497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.837810 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.837811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.838944 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.838946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.839856 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.839857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.839931 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.839945 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.840020 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.840071 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":25,"thread_start_us":67,"threads_started":1}
I20260924 13:42:48.840242 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.835351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:41805 (local address 127.0.0.1:45048)
0924 13:42:48.835540 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.835582 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.835647 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.836394 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.836395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.836397 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:48.836453 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.836454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.837769 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.837770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.839145 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.839148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.840008 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.840009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.840070 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.840074 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.840131 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.840170 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":38,"thread_start_us":50,"threads_started":1}
I20260924 13:42:48.840399 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.835759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.19.53.194:45089)
0924 13:42:48.835906 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.835911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.836304 (+ 393us) server_negotiation.cc:407] Connection header received
0924 13:42:48.836345 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.836347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.836363 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.836439 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:48.836576 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.837695 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.839033 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.839770 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.839847 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.839930 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.840261 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.840304 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":95,"threads_started":1}
I20260924 13:42:48.840421 27681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.835384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:39369 (local address 127.0.0.1:41076)
0924 13:42:48.835752 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.835760 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.835797 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.836169 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.836172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.836175 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:48.836247 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.836250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.838303 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.838304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.839344 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.839345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.840219 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.840221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.840286 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.840289 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.840319 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.840356 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":84,"threads_started":1}
I20260924 13:42:48.840834 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.835544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.19.53.195:39369)
0924 13:42:48.835982 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.835986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.835994 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:48.836035 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.836037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.836053 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.836525 (+ 472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:48.836581 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.838178 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.839442 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.840128 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.840616 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.840697 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.840720 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.840757 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":255,"threads_started":1}
I20260924 13:42:48.840857 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.835967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45048 (local address 127.19.53.193:41805)
0924 13:42:48.836226 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.836231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.836240 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:48.836275 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.836277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.836294 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.836523 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:48.836580 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.837668 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.839233 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.839945 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.840008 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.840072 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.840145 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.840182 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":76,"threads_started":1}
I20260924 13:42:48.841173 27448 tablet_service.cc:1467] Processing CreateTablet for tablet 0eca01a65454410d86c37d24c6fc3d4c (DEFAULT_TABLE table=test-table [id=4ebd6beb96594e24a34051d1afd176f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:48.841431 27583 tablet_service.cc:1467] Processing CreateTablet for tablet 0eca01a65454410d86c37d24c6fc3d4c (DEFAULT_TABLE table=test-table [id=4ebd6beb96594e24a34051d1afd176f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:48.841558 27448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eca01a65454410d86c37d24c6fc3d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:48.841463 27298 tablet_service.cc:1467] Processing CreateTablet for tablet 0eca01a65454410d86c37d24c6fc3d4c (DEFAULT_TABLE table=test-table [id=4ebd6beb96594e24a34051d1afd176f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:48.841758 27298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eca01a65454410d86c37d24c6fc3d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:48.841779 27583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eca01a65454410d86c37d24c6fc3d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:48.844730 27685 tablet_bootstrap.cc:492] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5: Bootstrap starting.
I20260924 13:42:48.845319 27685 tablet_bootstrap.cc:654] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:48.845577 27687 tablet_bootstrap.cc:492] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a: Bootstrap starting.
I20260924 13:42:48.845672 27685 log.cc:824] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:48.845772 27686 tablet_bootstrap.cc:492] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1: Bootstrap starting.
I20260924 13:42:48.846371 27687 tablet_bootstrap.cc:654] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:48.846396 27686 tablet_bootstrap.cc:654] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:48.846552 27685 tablet_bootstrap.cc:492] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5: No bootstrap required, opened a new log
I20260924 13:42:48.846632 27685 ts_tablet_manager.cc:1397] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:48.846648 27686 log.cc:824] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:48.846760 27687 log.cc:824] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:48.847311 27686 tablet_bootstrap.cc:492] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1: No bootstrap required, opened a new log
I20260924 13:42:48.847388 27686 ts_tablet_manager.cc:1397] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 13:42:48.847456 27687 tablet_bootstrap.cc:492] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a: No bootstrap required, opened a new log
I20260924 13:42:48.847525 27687 ts_tablet_manager.cc:1397] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:48.848702 27685 raft_consensus.cc:348] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } }
I20260924 13:42:48.848882 27685 raft_consensus.cc:374] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:48.848917 27685 raft_consensus.cc:729] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182cb6f20b2141ce96481e1f6f8658e5, State: Initialized, Role: FOLLOWER
I20260924 13:42:48.849066 27685 consensus_queue.cc:260] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } }
I20260924 13:42:48.849022 27686 raft_consensus.cc:348] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } }
I20260924 13:42:48.849169 27686 raft_consensus.cc:374] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:48.849212 27686 raft_consensus.cc:729] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4005cdf7604c09a87d7aed0a0be3f1, State: Initialized, Role: FOLLOWER
I20260924 13:42:48.849327 27685 ts_tablet_manager.cc:1428] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:42:48.849314 27686 consensus_queue.cc:260] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } }
I20260924 13:42:48.849449 27657 heartbeater.cc:503] Master 127.19.53.254:40635 was elected leader, sending a full tablet report...
I20260924 13:42:48.849584 27686 ts_tablet_manager.cc:1428] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:42:48.849612 27514 heartbeater.cc:503] Master 127.19.53.254:40635 was elected leader, sending a full tablet report...
I20260924 13:42:48.849812 27687 raft_consensus.cc:348] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } }
I20260924 13:42:48.849989 27687 raft_consensus.cc:374] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:48.850030 27687 raft_consensus.cc:729] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817228b4760b4a28b992d85ae1f3fc3a, State: Initialized, Role: FOLLOWER
I20260924 13:42:48.850157 27687 consensus_queue.cc:260] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } }
I20260924 13:42:48.850564 27374 heartbeater.cc:503] Master 127.19.53.254:40635 was elected leader, sending a full tablet report...
I20260924 13:42:48.850816 27687 ts_tablet_manager.cc:1428] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:42:48.854611 27693 raft_consensus.cc:482] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:48.854707 27693 raft_consensus.cc:504] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } }
I20260924 13:42:48.854990 27693 leader_election.cc:288] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195:39369), ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194:45089)
I20260924 13:42:48.857843 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.855164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:45089 (local address 127.19.53.193:33737)
0924 13:42:48.855227 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.855239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.855301 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.855469 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.855470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.855472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:48.855531 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.855533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.856174 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.856175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.857006 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.857008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.857540 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.857541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.857653 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.857655 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.857725 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.857781 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 13:42:48.857843 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.855295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:33737 (local address 127.19.53.194:45089)
0924 13:42:48.855361 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.855363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.855369 (+ 6us) server_negotiation.cc:407] Connection header received
0924 13:42:48.855386 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.855387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.855394 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.855465 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:48.855604 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.856102 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.857101 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.857451 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.857540 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.857560 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.857733 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.857783 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 13:42:48.858153 27468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eca01a65454410d86c37d24c6fc3d4c" candidate_uuid: "817228b4760b4a28b992d85ae1f3fc3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" is_pre_election: true
I20260924 13:42:48.858196 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.855164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:39369 (local address 127.19.53.193:34811)
0924 13:42:48.855477 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.855485 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.855557 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.855695 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.855696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.855697 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:48.855758 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.855760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.856491 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.856492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.857219 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.857220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.857938 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.857939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.858015 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.858019 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.858078 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.858115 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":155,"thread_start_us":106,"threads_started":1}
I20260924 13:42:48.858290 27468 raft_consensus.cc:2454] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 817228b4760b4a28b992d85ae1f3fc3a in term 0.
I20260924 13:42:48.858474 27261 leader_election.cc:302] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 817228b4760b4a28b992d85ae1f3fc3a, ab4005cdf7604c09a87d7aed0a0be3f1; no voters:
I20260924 13:42:48.858515 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.855222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:34811 (local address 127.19.53.195:39369)
0924 13:42:48.855276 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.855278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.855548 (+ 270us) server_negotiation.cc:407] Connection header received
0924 13:42:48.855606 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.855607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.855615 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.855690 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:48.855831 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.856404 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.857309 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.857800 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.857893 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.857902 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.858400 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.858451 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:42:48.858603 27693 raft_consensus.cc:2781] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:48.858642 27693 raft_consensus.cc:482] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:48.858666 27693 raft_consensus.cc:3037] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:48.858759 27610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eca01a65454410d86c37d24c6fc3d4c" candidate_uuid: "817228b4760b4a28b992d85ae1f3fc3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182cb6f20b2141ce96481e1f6f8658e5" is_pre_election: true
I20260924 13:42:48.858850 27610 raft_consensus.cc:2454] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 817228b4760b4a28b992d85ae1f3fc3a in term 0.
I20260924 13:42:48.859617 27693 raft_consensus.cc:504] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } }
I20260924 13:42:48.859745 27693 leader_election.cc:288] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [CANDIDATE]: Term 1 election: Requested vote from peers 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195:39369), ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194:45089)
I20260924 13:42:48.859845 27610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eca01a65454410d86c37d24c6fc3d4c" candidate_uuid: "817228b4760b4a28b992d85ae1f3fc3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182cb6f20b2141ce96481e1f6f8658e5"
I20260924 13:42:48.859854 27468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eca01a65454410d86c37d24c6fc3d4c" candidate_uuid: "817228b4760b4a28b992d85ae1f3fc3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1"
I20260924 13:42:48.859918 27610 raft_consensus.cc:3037] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:48.859941 27468 raft_consensus.cc:3037] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:48.860631 27468 raft_consensus.cc:2454] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 817228b4760b4a28b992d85ae1f3fc3a in term 1.
I20260924 13:42:48.860738 27610 raft_consensus.cc:2454] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 817228b4760b4a28b992d85ae1f3fc3a in term 1.
I20260924 13:42:48.860831 27261 leader_election.cc:302] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 817228b4760b4a28b992d85ae1f3fc3a, ab4005cdf7604c09a87d7aed0a0be3f1; no voters:
I20260924 13:42:48.860920 27693 raft_consensus.cc:2781] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:48.861081 27693 raft_consensus.cc:686] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [term 1 LEADER]: Becoming Leader. State: Replica: 817228b4760b4a28b992d85ae1f3fc3a, State: Running, Role: LEADER
I20260924 13:42:48.861163 27693 consensus_queue.cc:237] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } }
I20260924 13:42:48.861904 27169 catalog_manager.cc:5495] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 817228b4760b4a28b992d85ae1f3fc3a (127.19.53.193). New cstate: current_term: 1 leader_uuid: "817228b4760b4a28b992d85ae1f3fc3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } health_report { overall_health: UNKNOWN } } }
I20260924 13:42:48.871958 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.869562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.19.53.193:41805)
0924 13:42:48.869615 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.869618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.869637 (+ 19us) server_negotiation.cc:407] Connection header received
0924 13:42:48.869672 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.869672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.869687 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.869781 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:48.869922 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.870379 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.871162 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.871344 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.871447 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.871634 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:48.871731 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:48.871812 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.871863 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.871880 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 13:42:48.873617 27610 raft_consensus.cc:1261] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Refusing update from remote peer 817228b4760b4a28b992d85ae1f3fc3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:48.873617 27468 raft_consensus.cc:1261] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Refusing update from remote peer 817228b4760b4a28b992d85ae1f3fc3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:48.873941 27695 consensus_queue.cc:1035] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:48.874081 27693 consensus_queue.cc:1035] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:48.876433 27700 mvcc.cc:205] Tried to move back new op lower bound from 7332894182903894016 to 7332894182855876608. Current Snapshot: MvccSnapshot[applied={T|T < 7332894182903894016}]
I20260924 13:42:48.877065 27702 mvcc.cc:205] Tried to move back new op lower bound from 7332894182903894016 to 7332894182855876608. Current Snapshot: MvccSnapshot[applied={T|T < 7332894182903894016}]
I20260924 13:42:48.878585 27169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:48.878676 27169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:48.879420 27169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21: Sending DeleteTablet for 3 replicas of tablet 0eca01a65454410d86c37d24c6fc3d4c
I20260924 13:42:48.879738 27583 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eca01a65454410d86c37d24c6fc3d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:48 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:41076
I20260924 13:42:48.879747 27298 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eca01a65454410d86c37d24c6fc3d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:48 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:45048
I20260924 13:42:48.880003 27448 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eca01a65454410d86c37d24c6fc3d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:48 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:35076
I20260924 13:42:48.880028 27709 tablet_replica.cc:330] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5: stopping tablet replica
I20260924 13:42:48.880168 27709 raft_consensus.cc:2229] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:48.880268 27709 raft_consensus.cc:2258] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:48.880206 27169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:48.880336 27169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:48.880425 27710 tablet_replica.cc:330] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a: stopping tablet replica
I20260924 13:42:48.880508 27710 raft_consensus.cc:2229] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:42:48.880594 27710 raft_consensus.cc:2258] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:48.880692 27711 tablet_replica.cc:330] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1: stopping tablet replica
I20260924 13:42:48.880771 27711 raft_consensus.cc:2229] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:48.881047 27710 ts_tablet_manager.cc:1905] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:48.881112 27711 raft_consensus.cc:2258] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:48.881554 27448 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7778ba26f1454695e17e259ad947c3 (DEFAULT_TABLE table=test-table [id=f8d8ad58b06f48749a2b1d516b7f1ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:48.881659 27448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7778ba26f1454695e17e259ad947c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:48.881750 27583 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7778ba26f1454695e17e259ad947c3 (DEFAULT_TABLE table=test-table [id=f8d8ad58b06f48749a2b1d516b7f1ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:48.881875 27583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7778ba26f1454695e17e259ad947c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:48.882895 27685 tablet_bootstrap.cc:492] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5: Bootstrap starting.
I20260924 13:42:48.882952 27686 tablet_bootstrap.cc:492] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1: Bootstrap starting.
I20260924 13:42:48.883338 27686 tablet_bootstrap.cc:654] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:48.883352 27685 tablet_bootstrap.cc:654] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:48.883394 27709 ts_tablet_manager.cc:1905] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:48.883730 27298 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7778ba26f1454695e17e259ad947c3 (DEFAULT_TABLE table=test-table [id=f8d8ad58b06f48749a2b1d516b7f1ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:48.883826 27298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7778ba26f1454695e17e259ad947c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:48.884994 27709 ts_tablet_manager.cc:1918] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:48.885032 27687 tablet_bootstrap.cc:492] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a: Bootstrap starting.
I20260924 13:42:48.885121 27709 log.cc:1196] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/0eca01a65454410d86c37d24c6fc3d4c
I20260924 13:42:48.885325 27711 ts_tablet_manager.cc:1905] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:48.885417 27687 tablet_bootstrap.cc:654] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:48.885895 27709 ts_tablet_manager.cc:1939] T 0eca01a65454410d86c37d24c6fc3d4c P 182cb6f20b2141ce96481e1f6f8658e5: Deleting consensus metadata
I20260924 13:42:48.886353 27153 catalog_manager.cc:4841] TS 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195:39369): tablet 0eca01a65454410d86c37d24c6fc3d4c (table test-table [id=4ebd6beb96594e24a34051d1afd176f0]) successfully deleted
I20260924 13:42:48.886482 27710 ts_tablet_manager.cc:1918] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:48.886545 27710 log.cc:1196] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/wal/wals/0eca01a65454410d86c37d24c6fc3d4c
I20260924 13:42:48.886776 27711 ts_tablet_manager.cc:1918] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:48.886818 27710 ts_tablet_manager.cc:1939] T 0eca01a65454410d86c37d24c6fc3d4c P 817228b4760b4a28b992d85ae1f3fc3a: Deleting consensus metadata
I20260924 13:42:48.886902 27711 log.cc:1196] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/0eca01a65454410d86c37d24c6fc3d4c
I20260924 13:42:48.887001 27686 tablet_bootstrap.cc:492] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1: No bootstrap required, opened a new log
I20260924 13:42:48.887054 27686 ts_tablet_manager.cc:1397] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260924 13:42:48.887195 27156 catalog_manager.cc:4841] TS 817228b4760b4a28b992d85ae1f3fc3a (127.19.53.193:41805): tablet 0eca01a65454410d86c37d24c6fc3d4c (table test-table [id=4ebd6beb96594e24a34051d1afd176f0]) successfully deleted
I20260924 13:42:48.887215 27686 raft_consensus.cc:348] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.887284 27686 raft_consensus.cc:374] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:48.887305 27686 raft_consensus.cc:729] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4005cdf7604c09a87d7aed0a0be3f1, State: Initialized, Role: FOLLOWER
I20260924 13:42:48.887349 27686 consensus_queue.cc:260] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.887436 27686 ts_tablet_manager.cc:1428] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:48.887463 27711 ts_tablet_manager.cc:1939] T 0eca01a65454410d86c37d24c6fc3d4c P ab4005cdf7604c09a87d7aed0a0be3f1: Deleting consensus metadata
I20260924 13:42:48.887586 27685 tablet_bootstrap.cc:492] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5: No bootstrap required, opened a new log
I20260924 13:42:48.887634 27685 ts_tablet_manager.cc:1397] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 13:42:48.887773 27685 raft_consensus.cc:348] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.887842 27685 raft_consensus.cc:374] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:48.887866 27685 raft_consensus.cc:729] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182cb6f20b2141ce96481e1f6f8658e5, State: Initialized, Role: FOLLOWER
I20260924 13:42:48.887905 27685 consensus_queue.cc:260] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.888005 27685 ts_tablet_manager.cc:1428] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:48.888427 27156 catalog_manager.cc:4841] TS ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194:45089): tablet 0eca01a65454410d86c37d24c6fc3d4c (table test-table [id=4ebd6beb96594e24a34051d1afd176f0]) successfully deleted
I20260924 13:42:48.888437 27687 tablet_bootstrap.cc:492] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a: No bootstrap required, opened a new log
I20260924 13:42:48.888484 27687 ts_tablet_manager.cc:1397] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 13:42:48.888610 27687 raft_consensus.cc:348] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.888662 27687 raft_consensus.cc:374] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:48.888675 27687 raft_consensus.cc:729] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817228b4760b4a28b992d85ae1f3fc3a, State: Initialized, Role: FOLLOWER
I20260924 13:42:48.888731 27687 consensus_queue.cc:260] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.888827 27687 ts_tablet_manager.cc:1428] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260924 13:42:48.889801 27515 tablet.cc:2369] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:42:48.908793 27692 raft_consensus.cc:482] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:48.908895 27692 raft_consensus.cc:504] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.909224 27692 leader_election.cc:288] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 817228b4760b4a28b992d85ae1f3fc3a (127.19.53.193:41805), 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195:39369)
I20260924 13:42:48.912788 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.909527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:60433 (local address 127.19.53.193:41805)
0924 13:42:48.909703 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.909705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.909712 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:42:48.909735 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.909736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.909746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.909796 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:48.909999 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.910736 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.912051 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.912416 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.912503 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.912514 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.912679 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.912717 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260924 13:42:48.912809 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.909368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:41805 (local address 127.19.53.194:60433)
0924 13:42:48.909561 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.909573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.909619 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.909800 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.909802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.909803 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:48.909912 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.909914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.910812 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.910813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.911932 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.911933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.912525 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.912527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.912600 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.912605 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.912673 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.912712 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":9}
I20260924 13:42:48.912976 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.909387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:39369 (local address 127.19.53.194:50643)
0924 13:42:48.909564 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.909573 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.909619 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.909923 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.909924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.909925 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:48.909975 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.909976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.910717 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.910718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.911931 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.911933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.912710 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.912711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.912792 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.912795 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.912841 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.912888 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260924 13:42:48.913235 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.909715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:50643 (local address 127.19.53.195:39369)
0924 13:42:48.909780 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.909782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.909791 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:48.909817 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.909818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.909827 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.909889 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:48.910042 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.910639 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.912024 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.912598 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.912670 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.912678 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.913134 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.913176 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:42:48.913285 27327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7778ba26f1454695e17e259ad947c3" candidate_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817228b4760b4a28b992d85ae1f3fc3a" is_pre_election: true
I20260924 13:42:48.913465 27327 raft_consensus.cc:2454] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab4005cdf7604c09a87d7aed0a0be3f1 in term 0.
I20260924 13:42:48.913463 27610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7778ba26f1454695e17e259ad947c3" candidate_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182cb6f20b2141ce96481e1f6f8658e5" is_pre_election: true
I20260924 13:42:48.913565 27610 raft_consensus.cc:2454] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab4005cdf7604c09a87d7aed0a0be3f1 in term 0.
I20260924 13:42:48.913744 27402 leader_election.cc:302] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 817228b4760b4a28b992d85ae1f3fc3a, ab4005cdf7604c09a87d7aed0a0be3f1; no voters:
I20260924 13:42:48.913868 27692 raft_consensus.cc:2781] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:48.913902 27692 raft_consensus.cc:482] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:48.913947 27692 raft_consensus.cc:3037] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:48.914601 27692 raft_consensus.cc:504] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.914757 27692 leader_election.cc:288] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [CANDIDATE]: Term 1 election: Requested vote from peers 817228b4760b4a28b992d85ae1f3fc3a (127.19.53.193:41805), 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195:39369)
I20260924 13:42:48.914901 27610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7778ba26f1454695e17e259ad947c3" candidate_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182cb6f20b2141ce96481e1f6f8658e5"
I20260924 13:42:48.914950 27327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7778ba26f1454695e17e259ad947c3" candidate_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817228b4760b4a28b992d85ae1f3fc3a"
I20260924 13:42:48.914997 27610 raft_consensus.cc:3037] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:48.915017 27327 raft_consensus.cc:3037] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:48.915545 27610 raft_consensus.cc:2454] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab4005cdf7604c09a87d7aed0a0be3f1 in term 1.
I20260924 13:42:48.915712 27399 leader_election.cc:302] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182cb6f20b2141ce96481e1f6f8658e5, ab4005cdf7604c09a87d7aed0a0be3f1; no voters:
I20260924 13:42:48.915787 27692 raft_consensus.cc:2781] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:48.915831 27327 raft_consensus.cc:2454] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab4005cdf7604c09a87d7aed0a0be3f1 in term 1.
I20260924 13:42:48.915933 27692 raft_consensus.cc:686] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 LEADER]: Becoming Leader. State: Replica: ab4005cdf7604c09a87d7aed0a0be3f1, State: Running, Role: LEADER
I20260924 13:42:48.916006 27692 consensus_queue.cc:237] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.916631 27169 catalog_manager.cc:5495] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194). New cstate: current_term: 1 leader_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } health_report { overall_health: UNKNOWN } } }
Sep 24 13:42:48 dist-test-slave-tfm9 krb5kdc[27124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257368, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.254@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:48.950695 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.942834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.19.53.254:40635)
0924 13:42:48.942898 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:48.942900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:48.943020 (+ 120us) server_negotiation.cc:407] Connection header received
0924 13:42:48.943067 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:48.943068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:48.943079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:48.943174 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:48.943660 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.944427 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.945180 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.945455 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.945532 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.948035 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:48.948036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:48.948037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:48.948045 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:48.949425 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.949890 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.949891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.949892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.949995 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:48.950218 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:48.950218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:48.950219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:48.950322 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:48.950373 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:48.950557 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 13:42:48.950626 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 13:42:48.950803 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:48.942790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:40635 (local address 127.0.0.1:42032)
0924 13:42:48.942936 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:48.942949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:48.943018 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:48.943200 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:48.943201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:48.943508 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:48.943557 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.943559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.944516 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.944517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.945087 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:48.945088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:48.945536 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:48.945537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:48.945575 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:48.946188 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:48.946193 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:48.947939 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:48.949527 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.949536 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.949539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.949812 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.950074 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:48.950075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:48.950075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:48.950151 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:48.950391 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:48.950399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:48.950452 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 13:42:48.950532 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:42:48.950700 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":53,"thread_start_us":88,"threads_started":1}
I20260924 13:42:48.952945 27169 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:42032:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 13:42:48.953197 27169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:48.954694 27298 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0c664d2bc64e338549678e4401b7f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=226dd888b9c94e85aab9b9f742ff1abe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:42:48.954733 27583 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0c664d2bc64e338549678e4401b7f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=226dd888b9c94e85aab9b9f742ff1abe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:42:48.954862 27583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0c664d2bc64e338549678e4401b7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:48.954862 27298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0c664d2bc64e338549678e4401b7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:48.956329 27687 tablet_bootstrap.cc:492] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: Bootstrap starting.
I20260924 13:42:48.956328 27685 tablet_bootstrap.cc:492] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5: Bootstrap starting.
I20260924 13:42:48.956393 27448 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0c664d2bc64e338549678e4401b7f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=226dd888b9c94e85aab9b9f742ff1abe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:42:48.956526 27448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0c664d2bc64e338549678e4401b7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:48.956766 27685 tablet_bootstrap.cc:654] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:48.956784 27687 tablet_bootstrap.cc:654] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:48.957547 27687 tablet_bootstrap.cc:492] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: No bootstrap required, opened a new log
I20260924 13:42:48.957599 27687 ts_tablet_manager.cc:1397] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:48.957893 27687 raft_consensus.cc:348] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.958003 27687 raft_consensus.cc:374] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:48.958024 27687 raft_consensus.cc:729] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817228b4760b4a28b992d85ae1f3fc3a, State: Initialized, Role: FOLLOWER
I20260924 13:42:48.958068 27687 consensus_queue.cc:260] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.958124 27686 tablet_bootstrap.cc:492] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1: Bootstrap starting.
I20260924 13:42:48.958160 27687 ts_tablet_manager.cc:1428] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:48.958217 27705 tablet_replica.cc:439] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } }
I20260924 13:42:48.958279 27705 tablet_replica.cc:442] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:48.958547 27686 tablet_bootstrap.cc:654] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:48.958763 27685 tablet_bootstrap.cc:492] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5: No bootstrap required, opened a new log
I20260924 13:42:48.958797 27685 ts_tablet_manager.cc:1397] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:48.958941 27685 raft_consensus.cc:348] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.959012 27685 raft_consensus.cc:374] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:48.959035 27685 raft_consensus.cc:729] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182cb6f20b2141ce96481e1f6f8658e5, State: Initialized, Role: FOLLOWER
I20260924 13:42:48.959084 27685 consensus_queue.cc:260] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.959172 27685 ts_tablet_manager.cc:1428] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:48.959313 27691 tablet_replica.cc:439] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } }
I20260924 13:42:48.959430 27691 tablet_replica.cc:442] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:48.959750 27686 tablet_bootstrap.cc:492] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1: No bootstrap required, opened a new log
I20260924 13:42:48.959793 27686 ts_tablet_manager.cc:1397] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 13:42:48.960111 27686 raft_consensus.cc:348] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.960177 27686 raft_consensus.cc:374] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:48.960198 27686 raft_consensus.cc:729] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4005cdf7604c09a87d7aed0a0be3f1, State: Initialized, Role: FOLLOWER
I20260924 13:42:48.960245 27686 consensus_queue.cc:260] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:48.960332 27686 ts_tablet_manager.cc:1428] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 13:42:48.960381 27692 tablet_replica.cc:439] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } }
I20260924 13:42:48.960507 27692 tablet_replica.cc:442] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 13:42:48.988427 27375 tablet.cc:2369] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:42:49.041724 27658 tablet.cc:2369] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:42:49.254122 27705 raft_consensus.cc:482] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:49.254225 27705 raft_consensus.cc:504] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.254427 27705 leader_election.cc:288] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194:45089), 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195:39369)
I20260924 13:42:49.254709 27610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0c664d2bc64e338549678e4401b7f8" candidate_uuid: "817228b4760b4a28b992d85ae1f3fc3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182cb6f20b2141ce96481e1f6f8658e5" is_pre_election: true
I20260924 13:42:49.254796 27610 raft_consensus.cc:2454] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 817228b4760b4a28b992d85ae1f3fc3a in term 0.
I20260924 13:42:49.254721 27468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0c664d2bc64e338549678e4401b7f8" candidate_uuid: "817228b4760b4a28b992d85ae1f3fc3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" is_pre_election: true
I20260924 13:42:49.254846 27468 raft_consensus.cc:2454] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 817228b4760b4a28b992d85ae1f3fc3a in term 0.
I20260924 13:42:49.254983 27258 leader_election.cc:302] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182cb6f20b2141ce96481e1f6f8658e5, 817228b4760b4a28b992d85ae1f3fc3a; no voters:
I20260924 13:42:49.255095 27705 raft_consensus.cc:2781] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:49.255147 27705 raft_consensus.cc:482] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:49.255162 27705 raft_consensus.cc:3037] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:49.255859 27705 raft_consensus.cc:504] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.256004 27705 leader_election.cc:288] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [CANDIDATE]: Term 1 election: Requested vote from peers ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194:45089), 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195:39369)
I20260924 13:42:49.256140 27468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0c664d2bc64e338549678e4401b7f8" candidate_uuid: "817228b4760b4a28b992d85ae1f3fc3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1"
I20260924 13:42:49.256168 27610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0c664d2bc64e338549678e4401b7f8" candidate_uuid: "817228b4760b4a28b992d85ae1f3fc3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182cb6f20b2141ce96481e1f6f8658e5"
I20260924 13:42:49.256220 27610 raft_consensus.cc:3037] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:49.256237 27468 raft_consensus.cc:3037] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:49.256803 27610 raft_consensus.cc:2454] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 817228b4760b4a28b992d85ae1f3fc3a in term 1.
I20260924 13:42:49.256894 27468 raft_consensus.cc:2454] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 817228b4760b4a28b992d85ae1f3fc3a in term 1.
I20260924 13:42:49.257004 27258 leader_election.cc:302] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182cb6f20b2141ce96481e1f6f8658e5, 817228b4760b4a28b992d85ae1f3fc3a; no voters:
I20260924 13:42:49.257103 27705 raft_consensus.cc:2781] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:49.257165 27705 raft_consensus.cc:686] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [term 1 LEADER]: Becoming Leader. State: Replica: 817228b4760b4a28b992d85ae1f3fc3a, State: Running, Role: LEADER
I20260924 13:42:49.257225 27705 consensus_queue.cc:237] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.257500 27695 tablet_replica.cc:439] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: TxnStatusTablet state changed. Reason: New leader 817228b4760b4a28b992d85ae1f3fc3a. Latest consensus state: current_term: 1 leader_uuid: "817228b4760b4a28b992d85ae1f3fc3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } }
I20260924 13:42:49.257599 27695 tablet_replica.cc:442] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:42:49.257761 27728 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 13:42:49.257865 27728 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 13:42:49.257869 27169 catalog_manager.cc:5495] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 817228b4760b4a28b992d85ae1f3fc3a (127.19.53.193). New cstate: current_term: 1 leader_uuid: "817228b4760b4a28b992d85ae1f3fc3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260924 13:42:49.309245 27692 consensus_queue.cc:1035] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:49.311363 27731 consensus_queue.cc:1035] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:49.314927 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.312182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.19.53.193:41805)
0924 13:42:49.312221 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:49.312223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:49.312385 (+ 162us) server_negotiation.cc:407] Connection header received
0924 13:42:49.312407 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:49.312408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:49.312418 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:49.312464 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:49.312663 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.313244 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.314306 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.314483 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.314535 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.314725 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:49.314751 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:49.314788 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:49.314862 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 13:42:49.314877 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 13:42:49.314986 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.312124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:41805 (local address 127.0.0.1:45074)
0924 13:42:49.312254 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:49.312266 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:49.312316 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:49.312516 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:49.312517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:49.312518 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:42:49.312588 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.312591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.313316 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.313318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.314227 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.314231 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.314553 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.314554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.314629 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.314647 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:42:49.314803 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:42:49.314808 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:42:49.314861 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 13:42:49.314907 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 13:42:49.315894 27610 raft_consensus.cc:1261] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Refusing update from remote peer 817228b4760b4a28b992d85ae1f3fc3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:49.315950 27468 raft_consensus.cc:1261] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Refusing update from remote peer 817228b4760b4a28b992d85ae1f3fc3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:49.316140 27705 consensus_queue.cc:1035] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:49.316242 27695 consensus_queue.cc:1035] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:49.316576 27691 tablet_replica.cc:439] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5: TxnStatusTablet state changed. Reason: New leader 817228b4760b4a28b992d85ae1f3fc3a. Latest consensus state: current_term: 1 leader_uuid: "817228b4760b4a28b992d85ae1f3fc3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } }
I20260924 13:42:49.316668 27691 tablet_replica.cc:442] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:49.317024 27731 tablet_replica.cc:439] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1: TxnStatusTablet state changed. Reason: New leader 817228b4760b4a28b992d85ae1f3fc3a. Latest consensus state: current_term: 1 leader_uuid: "817228b4760b4a28b992d85ae1f3fc3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } }
I20260924 13:42:49.317117 27731 tablet_replica.cc:442] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:49.317492 27705 tablet_replica.cc:439] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "817228b4760b4a28b992d85ae1f3fc3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } }
I20260924 13:42:49.317569 27705 tablet_replica.cc:442] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:42:49.317986 27691 tablet_replica.cc:439] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "817228b4760b4a28b992d85ae1f3fc3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } }
I20260924 13:42:49.318073 27691 tablet_replica.cc:442] T 1f0c664d2bc64e338549678e4401b7f8 P 182cb6f20b2141ce96481e1f6f8658e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:49.318212 27705 tablet_replica.cc:439] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "817228b4760b4a28b992d85ae1f3fc3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } }
I20260924 13:42:49.318300 27705 tablet_replica.cc:442] T 1f0c664d2bc64e338549678e4401b7f8 P 817228b4760b4a28b992d85ae1f3fc3a: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:42:49.318524 27731 tablet_replica.cc:439] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "817228b4760b4a28b992d85ae1f3fc3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } } }
I20260924 13:42:49.318607 27731 tablet_replica.cc:442] T 1f0c664d2bc64e338549678e4401b7f8 P ab4005cdf7604c09a87d7aed0a0be3f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:42:49.322211 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.319585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.19.53.194:45089)
0924 13:42:49.319620 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:49.319622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:49.319629 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:42:49.319657 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:49.319658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:49.319680 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:49.319733 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:49.319871 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.320316 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.321351 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.321550 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.321649 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.321808 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:49.321967 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:49.322035 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:49.322091 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 13:42:49.322113 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:42:49.327050 27738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.324426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:41805 (local address 127.19.53.194:39513)
0924 13:42:49.324554 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:49.324565 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:49.324624 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:49.324746 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:49.324747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:49.324747 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:42:49.324793 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.324794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.325589 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.325590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.326226 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.326229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.326695 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.326696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.326742 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.326753 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:42:49.326926 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:42:49.326927 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:42:49.326960 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 13:42:49.326992 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":73,"threads_started":1}
I20260924 13:42:49.327061 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.324488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:39513 (local address 127.19.53.193:41805)
0924 13:42:49.324534 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:49.324535 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:49.324632 (+ 97us) server_negotiation.cc:407] Connection header received
0924 13:42:49.324673 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:49.324674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:49.324685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:49.324746 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:49.324833 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.325503 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.326320 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.326607 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.326681 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.326831 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:49.326863 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:49.326922 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:49.326978 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 13:42:49.326998 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 13:42:49.336215 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.333624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:33893 (local address 127.19.53.194:45089)
0924 13:42:49.333662 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:49.333663 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:49.333770 (+ 107us) server_negotiation.cc:407] Connection header received
0924 13:42:49.333842 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:49.333843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:49.333851 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:49.333890 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:49.334074 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.334534 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.335557 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.335757 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.335832 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.335986 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:49.336018 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:49.336070 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:49.336131 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 13:42:49.336149 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:42:49.336254 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.333555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:45089 (local address 127.19.53.193:33893)
0924 13:42:49.333698 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:49.333708 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:49.333786 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:49.333901 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:49.333902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:49.333903 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:42:49.333998 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.333999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.334626 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.334627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.335468 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.335470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.335831 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.335832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.335886 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.335900 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:42:49.336085 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:42:49.336086 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:42:49.336130 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 13:42:49.336173 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260924 13:42:49.349781 27168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:49.349900 27168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:49.350576 27168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21: Sending DeleteTablet for 3 replicas of tablet 9e7778ba26f1454695e17e259ad947c3
I20260924 13:42:49.350881 27298 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7778ba26f1454695e17e259ad947c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:49 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:45048
I20260924 13:42:49.350872 27583 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7778ba26f1454695e17e259ad947c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:49 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:41076
I20260924 13:42:49.350979 27710 tablet_replica.cc:330] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a: stopping tablet replica
I20260924 13:42:49.351032 27710 raft_consensus.cc:2229] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:49.351060 27710 raft_consensus.cc:2258] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:49.351118 27709 tablet_replica.cc:330] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5: stopping tablet replica
I20260924 13:42:49.351176 27709 raft_consensus.cc:2229] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:49.351217 27709 raft_consensus.cc:2258] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:49.351459 27448 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7778ba26f1454695e17e259ad947c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:49 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:35076
I20260924 13:42:49.351539 27711 tablet_replica.cc:330] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1: stopping tablet replica
I20260924 13:42:49.351563 27709 ts_tablet_manager.cc:1905] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:49.351578 27711 raft_consensus.cc:2229] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:42:49.351652 27711 raft_consensus.cc:2258] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:49.351929 27711 ts_tablet_manager.cc:1905] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:49.353456 27709 ts_tablet_manager.cc:1918] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:49.353523 27709 log.cc:1196] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/9e7778ba26f1454695e17e259ad947c3
I20260924 13:42:49.353538 27711 ts_tablet_manager.cc:1918] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:49.353582 27711 log.cc:1196] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/9e7778ba26f1454695e17e259ad947c3
I20260924 13:42:49.353740 27709 ts_tablet_manager.cc:1939] T 9e7778ba26f1454695e17e259ad947c3 P 182cb6f20b2141ce96481e1f6f8658e5: Deleting consensus metadata
I20260924 13:42:49.353780 27711 ts_tablet_manager.cc:1939] T 9e7778ba26f1454695e17e259ad947c3 P ab4005cdf7604c09a87d7aed0a0be3f1: Deleting consensus metadata
I20260924 13:42:49.354146 27153 catalog_manager.cc:4841] TS 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195:39369): tablet 9e7778ba26f1454695e17e259ad947c3 (table test-table [id=f8d8ad58b06f48749a2b1d516b7f1ff6]) successfully deleted
I20260924 13:42:49.354506 27156 catalog_manager.cc:4841] TS ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194:45089): tablet 9e7778ba26f1454695e17e259ad947c3 (table test-table [id=f8d8ad58b06f48749a2b1d516b7f1ff6]) successfully deleted
I20260924 13:42:49.357053 27710 ts_tablet_manager.cc:1905] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:49.358511 27710 ts_tablet_manager.cc:1918] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:49.358583 27710 log.cc:1196] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/wal/wals/9e7778ba26f1454695e17e259ad947c3
I20260924 13:42:49.358779 27710 ts_tablet_manager.cc:1939] T 9e7778ba26f1454695e17e259ad947c3 P 817228b4760b4a28b992d85ae1f3fc3a: Deleting consensus metadata
I20260924 13:42:49.359122 27156 catalog_manager.cc:4841] TS 817228b4760b4a28b992d85ae1f3fc3a (127.19.53.193:41805): tablet 9e7778ba26f1454695e17e259ad947c3 (table test-table [id=f8d8ad58b06f48749a2b1d516b7f1ff6]) successfully deleted
Sep 24 13:42:49 dist-test-slave-tfm9 krb5kdc[27124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 13:42:49 dist-test-slave-tfm9 krb5kdc[27124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:49 dist-test-slave-tfm9 krb5kdc[27124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 13:42:49.388062 27220 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:49.381083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.19.53.254:40635)
0924 13:42:49.381133 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:49.381135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:49.381284 (+ 149us) server_negotiation.cc:407] Connection header received
0924 13:42:49.381344 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:49.381345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:49.381364 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:49.381429 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:49.382133 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.382935 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.384035 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.384353 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.384435 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.387993 (+ 3558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42048: BlockingRecv error: recv got EOF from 127.0.0.1:42048 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 13:42:49 dist-test-slave-tfm9 krb5kdc[27124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM
I20260924 13:42:49.397089 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.389055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.19.53.254:40635)
0924 13:42:49.389091 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:49.389093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:49.389155 (+ 62us) server_negotiation.cc:407] Connection header received
0924 13:42:49.389216 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:49.389216 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:49.389226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:49.389276 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:49.389815 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.390565 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.391566 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.391811 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.391870 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.394413 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:49.394415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:49.394415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:49.394423 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:49.396063 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:49.396385 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:49.396386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:49.396386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:49.396454 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:49.396627 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:49.396628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:49.396629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:49.396753 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:49.396846 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:49.396921 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 13:42:49.396994 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:42:49.398702 27168 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:49.398846 27168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:49.400163 27298 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2a00135fc64862aa158b57a24bfa37 (DEFAULT_TABLE table=test-table [id=a51fd26ce04e4400a74a21b2aff4e778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:49.400188 27448 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2a00135fc64862aa158b57a24bfa37 (DEFAULT_TABLE table=test-table [id=a51fd26ce04e4400a74a21b2aff4e778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:49.400305 27298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2a00135fc64862aa158b57a24bfa37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:49.400303 27448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2a00135fc64862aa158b57a24bfa37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:49.401602 27687 tablet_bootstrap.cc:492] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a: Bootstrap starting.
I20260924 13:42:49.401664 27583 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2a00135fc64862aa158b57a24bfa37 (DEFAULT_TABLE table=test-table [id=a51fd26ce04e4400a74a21b2aff4e778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:49.401783 27583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2a00135fc64862aa158b57a24bfa37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:49.401979 27687 tablet_bootstrap.cc:654] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:49.402510 27687 tablet_bootstrap.cc:492] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a: No bootstrap required, opened a new log
I20260924 13:42:49.402572 27687 ts_tablet_manager.cc:1397] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:49.402683 27687 raft_consensus.cc:348] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.402769 27687 raft_consensus.cc:374] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:49.402796 27687 raft_consensus.cc:729] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817228b4760b4a28b992d85ae1f3fc3a, State: Initialized, Role: FOLLOWER
I20260924 13:42:49.402838 27687 consensus_queue.cc:260] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.402946 27687 ts_tablet_manager.cc:1428] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:49.403100 27686 tablet_bootstrap.cc:492] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1: Bootstrap starting.
I20260924 13:42:49.403508 27686 tablet_bootstrap.cc:654] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:49.403812 27685 tablet_bootstrap.cc:492] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5: Bootstrap starting.
I20260924 13:42:49.404197 27685 tablet_bootstrap.cc:654] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:49.404934 27686 tablet_bootstrap.cc:492] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1: No bootstrap required, opened a new log
I20260924 13:42:49.404982 27686 ts_tablet_manager.cc:1397] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 13:42:49.405104 27686 raft_consensus.cc:348] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.405172 27686 raft_consensus.cc:374] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:49.405200 27686 raft_consensus.cc:729] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4005cdf7604c09a87d7aed0a0be3f1, State: Initialized, Role: FOLLOWER
I20260924 13:42:49.405243 27686 consensus_queue.cc:260] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.405329 27686 ts_tablet_manager.cc:1428] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:49.405572 27685 tablet_bootstrap.cc:492] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5: No bootstrap required, opened a new log
I20260924 13:42:49.405644 27685 ts_tablet_manager.cc:1397] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:49.405787 27685 raft_consensus.cc:348] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.405854 27685 raft_consensus.cc:374] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:49.405874 27685 raft_consensus.cc:729] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182cb6f20b2141ce96481e1f6f8658e5, State: Initialized, Role: FOLLOWER
I20260924 13:42:49.406032 27685 consensus_queue.cc:260] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.406140 27685 ts_tablet_manager.cc:1428] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:49.489468 27695 raft_consensus.cc:482] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:49.489571 27695 raft_consensus.cc:504] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.489753 27695 leader_election.cc:288] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194:45089), 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195:39369)
I20260924 13:42:49.489997 27468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2a00135fc64862aa158b57a24bfa37" candidate_uuid: "817228b4760b4a28b992d85ae1f3fc3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" is_pre_election: true
I20260924 13:42:49.490043 27610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2a00135fc64862aa158b57a24bfa37" candidate_uuid: "817228b4760b4a28b992d85ae1f3fc3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182cb6f20b2141ce96481e1f6f8658e5" is_pre_election: true
I20260924 13:42:49.490103 27468 raft_consensus.cc:2454] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 817228b4760b4a28b992d85ae1f3fc3a in term 0.
I20260924 13:42:49.490137 27610 raft_consensus.cc:2454] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 817228b4760b4a28b992d85ae1f3fc3a in term 0.
I20260924 13:42:49.490268 27261 leader_election.cc:302] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 817228b4760b4a28b992d85ae1f3fc3a, ab4005cdf7604c09a87d7aed0a0be3f1; no voters:
I20260924 13:42:49.490358 27695 raft_consensus.cc:2781] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:49.490397 27695 raft_consensus.cc:482] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:49.490450 27695 raft_consensus.cc:3037] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:49.491110 27695 raft_consensus.cc:504] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.491243 27695 leader_election.cc:288] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [CANDIDATE]: Term 1 election: Requested vote from peers ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194:45089), 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195:39369)
I20260924 13:42:49.491389 27468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2a00135fc64862aa158b57a24bfa37" candidate_uuid: "817228b4760b4a28b992d85ae1f3fc3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1"
I20260924 13:42:49.491488 27468 raft_consensus.cc:3037] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:49.491483 27610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2a00135fc64862aa158b57a24bfa37" candidate_uuid: "817228b4760b4a28b992d85ae1f3fc3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182cb6f20b2141ce96481e1f6f8658e5"
I20260924 13:42:49.491564 27610 raft_consensus.cc:3037] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:49.492277 27468 raft_consensus.cc:2454] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 817228b4760b4a28b992d85ae1f3fc3a in term 1.
I20260924 13:42:49.492378 27610 raft_consensus.cc:2454] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 817228b4760b4a28b992d85ae1f3fc3a in term 1.
I20260924 13:42:49.492447 27261 leader_election.cc:302] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 817228b4760b4a28b992d85ae1f3fc3a, ab4005cdf7604c09a87d7aed0a0be3f1; no voters:
I20260924 13:42:49.492537 27695 raft_consensus.cc:2781] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:49.492581 27695 raft_consensus.cc:686] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [term 1 LEADER]: Becoming Leader. State: Replica: 817228b4760b4a28b992d85ae1f3fc3a, State: Running, Role: LEADER
I20260924 13:42:49.492633 27695 consensus_queue.cc:237] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.493172 27169 catalog_manager.cc:5495] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 817228b4760b4a28b992d85ae1f3fc3a (127.19.53.193). New cstate: current_term: 1 leader_uuid: "817228b4760b4a28b992d85ae1f3fc3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260924 13:42:49.519793 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.517297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.19.53.193:41805)
0924 13:42:49.517349 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:49.517351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:49.517362 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:42:49.517470 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:49.517470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:49.517480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:49.517541 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:49.517702 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.518247 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.519100 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.519291 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.519373 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.519528 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:49.519566 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:49.519627 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:49.519686 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 13:42:49.519704 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 13:42:49.520583 27610 raft_consensus.cc:1261] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Refusing update from remote peer 817228b4760b4a28b992d85ae1f3fc3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:49.520584 27468 raft_consensus.cc:1261] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Refusing update from remote peer 817228b4760b4a28b992d85ae1f3fc3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:49.520853 27693 consensus_queue.cc:1035] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:49.521018 27695 consensus_queue.cc:1035] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:49.522085 27698 mvcc.cc:205] Tried to move back new op lower bound from 7332894185554530304 to 7332894185442304000. Current Snapshot: MvccSnapshot[applied={T|T < 7332894185554530304}]
I20260924 13:42:49.523512 27168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:49.523586 27168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:49.524076 27168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21: Sending DeleteTablet for 3 replicas of tablet 5d2a00135fc64862aa158b57a24bfa37
I20260924 13:42:49.524327 27298 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2a00135fc64862aa158b57a24bfa37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:49 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:45048
I20260924 13:42:49.524358 27448 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2a00135fc64862aa158b57a24bfa37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:49 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:35076
I20260924 13:42:49.524420 27710 tablet_replica.cc:330] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a: stopping tablet replica
I20260924 13:42:49.524477 27710 raft_consensus.cc:2229] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:42:49.524492 27711 tablet_replica.cc:330] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1: stopping tablet replica
I20260924 13:42:49.524502 27583 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2a00135fc64862aa158b57a24bfa37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:49 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:41076
I20260924 13:42:49.524547 27710 raft_consensus.cc:2258] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:49.524549 27711 raft_consensus.cc:2229] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:49.524582 27709 tablet_replica.cc:330] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5: stopping tablet replica
I20260924 13:42:49.524598 27711 raft_consensus.cc:2258] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:49.524627 27709 raft_consensus.cc:2229] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:49.524679 27709 raft_consensus.cc:2258] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:49.524838 27168 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:42:49.524983 27168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:42:49.524978 27709 ts_tablet_manager.cc:1905] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:49.524981 27711 ts_tablet_manager.cc:1905] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:49.524982 27710 ts_tablet_manager.cc:1905] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:49.526417 27709 ts_tablet_manager.cc:1918] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:49.526434 27710 ts_tablet_manager.cc:1918] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:49.526417 27711 ts_tablet_manager.cc:1918] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:42:49.526485 27709 log.cc:1196] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/5d2a00135fc64862aa158b57a24bfa37
I20260924 13:42:49.526504 27711 log.cc:1196] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/5d2a00135fc64862aa158b57a24bfa37
I20260924 13:42:49.526506 27710 log.cc:1196] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/wal/wals/5d2a00135fc64862aa158b57a24bfa37
I20260924 13:42:49.526710 27709 ts_tablet_manager.cc:1939] T 5d2a00135fc64862aa158b57a24bfa37 P 182cb6f20b2141ce96481e1f6f8658e5: Deleting consensus metadata
I20260924 13:42:49.526722 27711 ts_tablet_manager.cc:1939] T 5d2a00135fc64862aa158b57a24bfa37 P ab4005cdf7604c09a87d7aed0a0be3f1: Deleting consensus metadata
I20260924 13:42:49.526765 27710 ts_tablet_manager.cc:1939] T 5d2a00135fc64862aa158b57a24bfa37 P 817228b4760b4a28b992d85ae1f3fc3a: Deleting consensus metadata
I20260924 13:42:49.527113 27156 catalog_manager.cc:4841] TS 817228b4760b4a28b992d85ae1f3fc3a (127.19.53.193:41805): tablet 5d2a00135fc64862aa158b57a24bfa37 (table test-table [id=a51fd26ce04e4400a74a21b2aff4e778]) successfully deleted
I20260924 13:42:49.527145 27153 catalog_manager.cc:4841] TS 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195:39369): tablet 5d2a00135fc64862aa158b57a24bfa37 (table test-table [id=a51fd26ce04e4400a74a21b2aff4e778]) successfully deleted
I20260924 13:42:49.527191 27156 catalog_manager.cc:4841] TS ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194:45089): tablet 5d2a00135fc64862aa158b57a24bfa37 (table test-table [id=a51fd26ce04e4400a74a21b2aff4e778]) successfully deleted
I20260924 13:42:49.527947 27298 tablet_service.cc:1467] Processing CreateTablet for tablet fb39bafd61bb41b7b205b21a945336ed (DEFAULT_TABLE table=test-table [id=19d4db044add4bcabac372da4a7d271e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:49.527989 27583 tablet_service.cc:1467] Processing CreateTablet for tablet fb39bafd61bb41b7b205b21a945336ed (DEFAULT_TABLE table=test-table [id=19d4db044add4bcabac372da4a7d271e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:49.528062 27298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb39bafd61bb41b7b205b21a945336ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:49.528124 27583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb39bafd61bb41b7b205b21a945336ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:49.528925 27687 tablet_bootstrap.cc:492] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a: Bootstrap starting.
I20260924 13:42:49.528927 27685 tablet_bootstrap.cc:492] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5: Bootstrap starting.
I20260924 13:42:49.529165 27448 tablet_service.cc:1467] Processing CreateTablet for tablet fb39bafd61bb41b7b205b21a945336ed (DEFAULT_TABLE table=test-table [id=19d4db044add4bcabac372da4a7d271e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:42:49.529222 27687 tablet_bootstrap.cc:654] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:49.529291 27448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb39bafd61bb41b7b205b21a945336ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:49.529606 27685 tablet_bootstrap.cc:654] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:49.529758 27687 tablet_bootstrap.cc:492] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a: No bootstrap required, opened a new log
I20260924 13:42:49.529817 27687 ts_tablet_manager.cc:1397] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:49.529953 27687 raft_consensus.cc:348] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.530042 27687 raft_consensus.cc:374] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:49.530069 27687 raft_consensus.cc:729] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817228b4760b4a28b992d85ae1f3fc3a, State: Initialized, Role: FOLLOWER
I20260924 13:42:49.530115 27687 consensus_queue.cc:260] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.530210 27687 ts_tablet_manager.cc:1428] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:49.530577 27686 tablet_bootstrap.cc:492] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1: Bootstrap starting.
I20260924 13:42:49.530859 27686 tablet_bootstrap.cc:654] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:49.530900 27685 tablet_bootstrap.cc:492] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5: No bootstrap required, opened a new log
I20260924 13:42:49.530961 27685 ts_tablet_manager.cc:1397] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:42:49.531105 27685 raft_consensus.cc:348] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.531178 27685 raft_consensus.cc:374] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:49.531200 27685 raft_consensus.cc:729] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182cb6f20b2141ce96481e1f6f8658e5, State: Initialized, Role: FOLLOWER
I20260924 13:42:49.531251 27685 consensus_queue.cc:260] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.531343 27685 ts_tablet_manager.cc:1428] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:49.531435 27686 tablet_bootstrap.cc:492] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1: No bootstrap required, opened a new log
I20260924 13:42:49.531507 27686 ts_tablet_manager.cc:1397] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:49.531612 27686 raft_consensus.cc:348] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.531685 27686 raft_consensus.cc:374] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:49.531706 27686 raft_consensus.cc:729] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4005cdf7604c09a87d7aed0a0be3f1, State: Initialized, Role: FOLLOWER
I20260924 13:42:49.531759 27686 consensus_queue.cc:260] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.531862 27686 ts_tablet_manager.cc:1428] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:49.637789 27691 raft_consensus.cc:482] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:42:49.637928 27691 raft_consensus.cc:504] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.638301 27691 leader_election.cc:288] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 817228b4760b4a28b992d85ae1f3fc3a (127.19.53.193:41805), ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194:45089)
I20260924 13:42:49.641795 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.638691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:45089 (local address 127.19.53.195:48467)
0924 13:42:49.638860 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:49.638872 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:49.638943 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:49.639093 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:49.639094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:49.639095 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:49.639165 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.639166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.640039 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.640040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.640890 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.640893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.641531 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.641532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.641615 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.641620 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:42:49.641675 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 13:42:49.641720 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
I20260924 13:42:49.641810 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.638836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:48467 (local address 127.19.53.194:45089)
0924 13:42:49.638886 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:49.638888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:49.638924 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:42:49.639024 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:49.639025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:49.639035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:49.639084 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:49.639236 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.639924 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.640988 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.641434 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.641504 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.641512 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:49.641696 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 13:42:49.641733 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 13:42:49.642040 27468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb39bafd61bb41b7b205b21a945336ed" candidate_uuid: "182cb6f20b2141ce96481e1f6f8658e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" is_pre_election: true
I20260924 13:42:49.642143 27468 raft_consensus.cc:2454] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 182cb6f20b2141ce96481e1f6f8658e5 in term 0.
I20260924 13:42:49.642280 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.638601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:45797 (local address 127.19.53.193:41805)
0924 13:42:49.638655 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:49.638657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:49.638737 (+ 80us) server_negotiation.cc:407] Connection header received
0924 13:42:49.638832 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:49.638834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:49.638845 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:49.638875 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:42:49.639062 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.639860 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.641245 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.641765 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.641829 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.641840 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:49.642165 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 13:42:49.642206 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 13:42:49.642325 27543 leader_election.cc:302] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182cb6f20b2141ce96481e1f6f8658e5, ab4005cdf7604c09a87d7aed0a0be3f1; no voters:
I20260924 13:42:49.642478 27691 raft_consensus.cc:2781] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:42:49.642498 27765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.638512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:41805 (local address 127.19.53.195:45797)
0924 13:42:49.638650 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:49.638662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:49.638742 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:49.638946 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:49.638947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:49.638949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:42:49.639004 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.639006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.639956 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.639957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.641150 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.641153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.641981 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.641981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.642063 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.642067 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:42:49.642363 (+ 296us) client_negotiation.cc:240] Negotiation successful
0924 13:42:49.642405 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260924 13:42:49.642534 27691 raft_consensus.cc:482] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:42:49.642558 27691 raft_consensus.cc:3037] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:49.643205 27691 raft_consensus.cc:504] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.643287 27691 leader_election.cc:288] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [CANDIDATE]: Term 1 election: Requested vote from peers 817228b4760b4a28b992d85ae1f3fc3a (127.19.53.193:41805), ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194:45089)
I20260924 13:42:49.643565 27468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb39bafd61bb41b7b205b21a945336ed" candidate_uuid: "182cb6f20b2141ce96481e1f6f8658e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1"
I20260924 13:42:49.643590 27327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb39bafd61bb41b7b205b21a945336ed" candidate_uuid: "182cb6f20b2141ce96481e1f6f8658e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817228b4760b4a28b992d85ae1f3fc3a" is_pre_election: true
I20260924 13:42:49.643604 27326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb39bafd61bb41b7b205b21a945336ed" candidate_uuid: "182cb6f20b2141ce96481e1f6f8658e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817228b4760b4a28b992d85ae1f3fc3a"
I20260924 13:42:49.643662 27468 raft_consensus.cc:3037] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:49.643693 27327 raft_consensus.cc:2454] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 182cb6f20b2141ce96481e1f6f8658e5 in term 0.
I20260924 13:42:49.644306 27468 raft_consensus.cc:2454] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 182cb6f20b2141ce96481e1f6f8658e5 in term 1.
I20260924 13:42:49.644521 27543 leader_election.cc:302] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 182cb6f20b2141ce96481e1f6f8658e5, ab4005cdf7604c09a87d7aed0a0be3f1; no voters: 817228b4760b4a28b992d85ae1f3fc3a
I20260924 13:42:49.644611 27691 raft_consensus.cc:2781] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:49.644719 27691 raft_consensus.cc:686] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 LEADER]: Becoming Leader. State: Replica: 182cb6f20b2141ce96481e1f6f8658e5, State: Running, Role: LEADER
I20260924 13:42:49.644814 27691 consensus_queue.cc:237] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } }
I20260924 13:42:49.645475 27168 catalog_manager.cc:5495] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195). New cstate: current_term: 1 leader_uuid: "182cb6f20b2141ce96481e1f6f8658e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "182cb6f20b2141ce96481e1f6f8658e5" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39369 } health_report { overall_health: HEALTHY } } }
I20260924 13:42:49.680154 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.677969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.19.53.195:39369)
0924 13:42:49.678095 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:49.678097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:49.678104 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:42:49.678139 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:49.678140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:49.678158 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:49.678229 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:49.678340 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.678837 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.679503 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.679698 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.679745 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.679860 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:49.679968 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:49.680017 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:49.680053 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 13:42:49.680067 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260924 13:42:49.684544 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.682557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:43473 (local address 127.19.53.193:41805)
0924 13:42:49.682607 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:49.682608 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:49.682636 (+ 28us) server_negotiation.cc:407] Connection header received
0924 13:42:49.682694 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:49.682695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:49.682703 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:49.682740 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:49.682841 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.683263 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.683955 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.684132 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.684178 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.684364 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:49.684388 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:49.684425 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:49.684479 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 13:42:49.684495 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 13:42:49.684573 27772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.682417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:41805 (local address 127.19.53.195:43473)
0924 13:42:49.682578 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:49.682589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:49.682647 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:49.682753 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:49.682754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:49.682755 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:42:49.682803 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.682804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.683317 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.683317 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.683869 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.683870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.684204 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.684204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.684275 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.684290 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:42:49.684442 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:42:49.684444 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:42:49.684478 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 13:42:49.684507 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260924 13:42:49.686564 27326 raft_consensus.cc:3037] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:49.686600 27468 raft_consensus.cc:1261] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Refusing update from remote peer 182cb6f20b2141ce96481e1f6f8658e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:49.686815 27691 consensus_queue.cc:1035] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab4005cdf7604c09a87d7aed0a0be3f1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 45089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:49.687664 27326 raft_consensus.cc:1261] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Refusing update from remote peer 182cb6f20b2141ce96481e1f6f8658e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:49.687840 27691 consensus_queue.cc:1035] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "817228b4760b4a28b992d85ae1f3fc3a" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:49.694003 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.691739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:54145 (local address 127.19.53.195:39369)
0924 13:42:49.691773 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:49.691775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:49.691783 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:49.691844 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:49.691845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:49.691852 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:49.691912 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:49.692052 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.692574 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.693236 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.693440 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.693543 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.693702 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:49.693731 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:49.693798 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:49.693898 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 13:42:49.693940 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:42:49.694006 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:49.691694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:39369 (local address 127.19.53.193:54145)
0924 13:42:49.691734 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:49.691745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:49.691788 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:49.691936 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:49.691937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:49.691939 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:42:49.691977 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.691978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.692625 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.692625 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.693170 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:49.693172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:49.693555 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:49.693556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:49.693627 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:49.693634 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:42:49.693825 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:42:49.693828 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:42:49.693869 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 13:42:49.693941 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 13:42:49.700596 27168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:49.700692 27168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:42:49.701323 27168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 936aaeea5ba74a48aebb0587c9329e21: Sending DeleteTablet for 3 replicas of tablet fb39bafd61bb41b7b205b21a945336ed
I20260924 13:42:49.701545 27583 tablet_service.cc:1514] Processing DeleteTablet for tablet fb39bafd61bb41b7b205b21a945336ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:49 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:41076
I20260924 13:42:49.701591 27448 tablet_service.cc:1514] Processing DeleteTablet for tablet fb39bafd61bb41b7b205b21a945336ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:49 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:35076
I20260924 13:42:49.701623 27709 tablet_replica.cc:330] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5: stopping tablet replica
I20260924 13:42:49.701624 27298 tablet_service.cc:1514] Processing DeleteTablet for tablet fb39bafd61bb41b7b205b21a945336ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:42:49 UTC) from {username='oryx', principal='oryx/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:45048
I20260924 13:42:49.701665 27711 tablet_replica.cc:330] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1: stopping tablet replica
I20260924 13:42:49.701663 27709 raft_consensus.cc:2229] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:42:49.701705 27711 raft_consensus.cc:2229] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:49.701725 27709 raft_consensus.cc:2258] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:49.701735 27711 raft_consensus.cc:2258] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:49.701803 27710 tablet_replica.cc:330] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a: stopping tablet replica
I20260924 13:42:49.701869 27710 raft_consensus.cc:2229] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:42:49.701924 27710 raft_consensus.cc:2258] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:42:49.702160 27711 ts_tablet_manager.cc:1905] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:49.702234 27709 ts_tablet_manager.cc:1905] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:49.703477 27710 ts_tablet_manager.cc:1905] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:42:49.703859 27709 ts_tablet_manager.cc:1918] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:49.703868 27711 ts_tablet_manager.cc:1918] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:49.703917 27711 log.cc:1196] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/fb39bafd61bb41b7b205b21a945336ed
I20260924 13:42:49.703918 27709 log.cc:1196] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/fb39bafd61bb41b7b205b21a945336ed
I20260924 13:42:49.704160 27711 ts_tablet_manager.cc:1939] T fb39bafd61bb41b7b205b21a945336ed P ab4005cdf7604c09a87d7aed0a0be3f1: Deleting consensus metadata
I20260924 13:42:49.704165 27709 ts_tablet_manager.cc:1939] T fb39bafd61bb41b7b205b21a945336ed P 182cb6f20b2141ce96481e1f6f8658e5: Deleting consensus metadata
I20260924 13:42:49.704475 27156 catalog_manager.cc:4841] TS ab4005cdf7604c09a87d7aed0a0be3f1 (127.19.53.194:45089): tablet fb39bafd61bb41b7b205b21a945336ed (table test-table [id=19d4db044add4bcabac372da4a7d271e]) successfully deleted
I20260924 13:42:49.705050 27710 ts_tablet_manager.cc:1918] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:42:49.705121 27710 log.cc:1196] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257335418480-19671-0/minicluster-data/ts-0/wal/wals/fb39bafd61bb41b7b205b21a945336ed
I20260924 13:42:49.705324 27710 ts_tablet_manager.cc:1939] T fb39bafd61bb41b7b205b21a945336ed P 817228b4760b4a28b992d85ae1f3fc3a: Deleting consensus metadata
I20260924 13:42:49.705633 27153 catalog_manager.cc:4841] TS 182cb6f20b2141ce96481e1f6f8658e5 (127.19.53.195:39369): tablet fb39bafd61bb41b7b205b21a945336ed (table test-table [id=19d4db044add4bcabac372da4a7d271e]) successfully deleted
I20260924 13:42:49.705713 27156 catalog_manager.cc:4841] TS 817228b4760b4a28b992d85ae1f3fc3a (127.19.53.193:41805): tablet fb39bafd61bb41b7b205b21a945336ed (table test-table [id=19d4db044add4bcabac372da4a7d271e]) successfully deleted
I20260924 13:42:49.705987 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 27242
I20260924 13:42:49.713773 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 27383
I20260924 13:42:49.719491 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 27524
I20260924 13:42:49.724095 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 27137
2026-09-24T13:42:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2705 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:49 dist-test-slave-tfm9 krb5kdc[27779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:49 dist-test-slave-tfm9 krb5kdc[27779](info): set up 2 sockets
Sep 24 13:42:49 dist-test-slave-tfm9 krb5kdc[27779](info): commencing operation
krb5kdc: starting...
W20260924 13:42:50.756081 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:50 dist-test-slave-tfm9 krb5kdc[27779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:42:50.868933 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:37187
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:37187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:50.948974 27792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:50.949208 27792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:50.949245 27792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:50.950470 27792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:50.950533 27792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:50.950557 27792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:50.950587 27792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:50.950619 27792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:50.952095 27792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:37187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:37187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:50.952373 27792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:50.952603 27792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:42:50.957372 27792 server_base.cc:1034] running on GCE node
W20260924 13:42:50.957297 27797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:50.957319 27798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:50.957297 27800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:50.957825 27792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:50.958195 27792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:50.959336 27792 hybrid_clock.cc:648] HybridClock initialized: now 1790257370959316 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T13:42:50Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1241 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:51 dist-test-slave-tfm9 krb5kdc[27805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:51 dist-test-slave-tfm9 krb5kdc[27805](info): set up 2 sockets
Sep 24 13:42:51 dist-test-slave-tfm9 krb5kdc[27805](info): commencing operation
krb5kdc: starting...
W20260924 13:42:52.004004 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:42:52 dist-test-slave-tfm9 krb5kdc[27805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:42:52.118276 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:38999
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:38999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:52.199484 27818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:52.199697 27818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:52.199721 27818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:52.200909 27818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:52.200968 27818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:52.200984 27818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:52.200997 27818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:52.201031 27818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:52.202632 27818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:38999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:38999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:52.202853 27818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:52.203061 27818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:52.207938 27824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:52.207947 27826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:52.208016 27823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:52.208258 27818 server_base.cc:1034] running on GCE node
I20260924 13:42:52.208546 27818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:52.208801 27818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:52.210037 27818 hybrid_clock.cc:648] HybridClock initialized: now 1790257372210027 us; error 29 us; skew 500 ppm
Sep 24 13:42:52 dist-test-slave-tfm9 krb5kdc[27805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257372, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:52.216943 27818 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:42:52.217298 27818 webserver.cc:460] Webserver started at http://127.19.53.254:35777/ using document root <none> and password file <none>
I20260924 13:42:52.217506 27818 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:52.217547 27818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:52.217631 27818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:52.218694 27818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "d5bd2fdade1558d9a3ed2db2c0c9cbc1"
server_key_iv: "3cfd82995c9d51470c403c46c1555e24"
server_key_version: "encryptionkey@0"
I20260924 13:42:52.218988 27818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "d5bd2fdade1558d9a3ed2db2c0c9cbc1"
server_key_iv: "3cfd82995c9d51470c403c46c1555e24"
server_key_version: "encryptionkey@0"
I20260924 13:42:52.220223 27818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:52.220937 27833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:52.221129 27818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:52.221210 27818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "d5bd2fdade1558d9a3ed2db2c0c9cbc1"
server_key_iv: "3cfd82995c9d51470c403c46c1555e24"
server_key_version: "encryptionkey@0"
I20260924 13:42:52.221259 27818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:52.248795 27818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:52.251091 27818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:52.251199 27818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:52.254873 27818 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:38999
I20260924 13:42:52.254907 27886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:38999 every 8 connection(s)
I20260924 13:42:52.255254 27818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:52.256150 27887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:52.258469 27887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b: Bootstrap starting.
I20260924 13:42:52.259043 27887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:52.259330 27887 log.cc:824] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:52.260128 27887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b: No bootstrap required, opened a new log
I20260924 13:42:52.261675 27887 raft_consensus.cc:348] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38999 } }
I20260924 13:42:52.261809 27887 raft_consensus.cc:374] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:52.261831 27887 raft_consensus.cc:729] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77c3edcd81c4f0c8f82783bc6d6e56b, State: Initialized, Role: FOLLOWER
I20260924 13:42:52.262032 27887 consensus_queue.cc:260] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38999 } }
I20260924 13:42:52.262096 27887 raft_consensus.cc:388] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:52.262118 27887 raft_consensus.cc:482] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:52.262188 27887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:52.262689 27887 raft_consensus.cc:504] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38999 } }
I20260924 13:42:52.262770 27887 leader_election.cc:302] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a77c3edcd81c4f0c8f82783bc6d6e56b; no voters:
I20260924 13:42:52.262945 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 27818
I20260924 13:42:52.262970 27887 leader_election.cc:288] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:52.262998 27890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:52.263057 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:42:52.263121 27890 raft_consensus.cc:686] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 1 LEADER]: Becoming Leader. State: Replica: a77c3edcd81c4f0c8f82783bc6d6e56b, State: Running, Role: LEADER
I20260924 13:42:52.263219 27887 sys_catalog.cc:564] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:52.263209 27890 consensus_queue.cc:237] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38999 } }
I20260924 13:42:52.263247 19671 external_mini_cluster.cc:1428] Setting key ff9705f0f43f72f389c70798eae3e1eb
I20260924 13:42:52.263513 27890 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38999 } } }
I20260924 13:42:52.263584 27890 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:52.263705 27891 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a77c3edcd81c4f0c8f82783bc6d6e56b. Latest consensus state: current_term: 1 leader_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38999 } } }
I20260924 13:42:52.263808 27891 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:52.263886 27895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:52.264405 27895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:52.266125 27895 catalog_manager.cc:1347] Generated new cluster ID: 448e043a033f4b0a87abb1beebc02b63
I20260924 13:42:52.266178 27895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:52.270989 27895 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:52.271617 27895 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 13:42:52 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:52.275990 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:52.263973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.19.53.254:38999)
0924 13:42:52.264251 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:52.264261 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:52.264982 (+ 721us) server_negotiation.cc:407] Connection header received
0924 13:42:52.265098 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:52.265106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:52.265172 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:52.265317 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:52.266273 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.267662 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.268781 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.269392 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.269491 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:52.272426 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:52.272430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:52.272431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:52.272443 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:52.274443 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:52.274807 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:52.274810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:52.274810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:52.274917 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:52.275112 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:52.275113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:52.275113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:52.275385 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:52.275556 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:52.275620 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 13:42:52.275765 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":81,"threads_started":1}
I20260924 13:42:52.278908 27895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b: Generated new TSK 0
I20260924 13:42:52.279068 27895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:42:52.308264 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38999
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:52.389494 27913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:52.389739 27913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:52.389777 27913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:52.390987 27913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:52.391057 27913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:52.391103 27913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:52.392524 27913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:52.392808 27913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:52.393031 27913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:52.397958 27919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:52.398034 27918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:52.398013 27921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:52.398129 27913 server_base.cc:1034] running on GCE node
I20260924 13:42:52.398376 27913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:52.398639 27913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:52.399775 27913 hybrid_clock.cc:648] HybridClock initialized: now 1790257372399760 us; error 40 us; skew 500 ppm
Sep 24 13:42:52 dist-test-slave-tfm9 krb5kdc[27805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257372, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:52.402208 27913 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:42:52.402634 27913 webserver.cc:460] Webserver started at http://127.19.53.193:40349/ using document root <none> and password file <none>
I20260924 13:42:52.402835 27913 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:52.402878 27913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:52.402971 27913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:52.403960 27913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "37389a2770b74236b53312ef250af8ea"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "243a6b99ac423d30d624f73f45f0e198"
server_key_iv: "4b7d8c74b1b05c36ccfb1d50042621ad"
server_key_version: "encryptionkey@0"
I20260924 13:42:52.404337 27913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "37389a2770b74236b53312ef250af8ea"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "243a6b99ac423d30d624f73f45f0e198"
server_key_iv: "4b7d8c74b1b05c36ccfb1d50042621ad"
server_key_version: "encryptionkey@0"
I20260924 13:42:52.405619 27913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:42:52.406524 27928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:52.406697 27913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:52.406776 27913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "37389a2770b74236b53312ef250af8ea"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "243a6b99ac423d30d624f73f45f0e198"
server_key_iv: "4b7d8c74b1b05c36ccfb1d50042621ad"
server_key_version: "encryptionkey@0"
I20260924 13:42:52.406827 27913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:52.417486 27913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:52.419329 27913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:52.419435 27913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:52.419601 27913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:52.419948 27913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:52.419984 27913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:52.420033 27913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:52.420056 27913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:52.426555 27913 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:35809
I20260924 13:42:52.426659 28042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:35809 every 8 connection(s)
I20260924 13:42:52.426918 27913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:42:52.432821 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 27913
I20260924 13:42:52.432912 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:42:52.433121 19671 external_mini_cluster.cc:1428] Setting key 0e1041b38668171afc0edd156fdacbb2
Sep 24 13:42:52 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257372, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:52.437402 28045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:52.427560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:49631)
0924 13:42:52.427723 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:52.427766 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:52.428459 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:52.428723 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:52.428727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:52.429448 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:52.429661 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:52.429665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.430695 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.430697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:52.431523 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:52.431526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.431999 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.432001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:52.432122 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:52.432635 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:52.432643 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:52.434158 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:52.435899 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:52.435907 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:52.435909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:52.436219 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:52.436463 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:52.436464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:52.436464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:52.436600 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:52.436863 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:52.436868 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:52.437041 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 13:42:52.437142 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 13:42:52.437254 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260924 13:42:52.437410 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:52.427811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:49631 (local address 127.19.53.254:38999)
0924 13:42:52.427859 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:52.427861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:52.427912 (+ 51us) server_negotiation.cc:407] Connection header received
0924 13:42:52.428574 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:52.428576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:52.428588 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:52.428647 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:52.429775 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.430606 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.431687 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.431898 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.431981 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:52.434254 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:52.434256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:52.434256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:52.434264 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:52.435810 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:52.436316 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:52.436317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:52.436317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:52.436387 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:52.436683 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:52.436684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:52.436685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:52.436793 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:52.436849 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:52.437188 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 13:42:52.437282 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:42:52.437896 28043 heartbeater.cc:347] Connected to a master server at 127.19.53.254:38999
I20260924 13:42:52.438023 28043 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:52.438189 28043 heartbeater.cc:511] Master 127.19.53.254:38999 requested a full tablet report, sending...
I20260924 13:42:52.438753 27851 ts_manager.cc:194] Registered new tserver with Master: 37389a2770b74236b53312ef250af8ea (127.19.53.193:35809)
I20260924 13:42:52.439834 27851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:49631
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:42:52.458933 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38999
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:52.539263 28048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:52.539460 28048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:52.539482 28048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:52.540539 28048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:52.540570 28048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:52.540593 28048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:52.541868 28048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:52.542065 28048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:52.542224 28048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:52.547077 28056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:52.547168 28054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:52.547077 28053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:52.547154 28048 server_base.cc:1034] running on GCE node
I20260924 13:42:52.547519 28048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:52.547709 28048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:52.548848 28048 hybrid_clock.cc:648] HybridClock initialized: now 1790257372548835 us; error 33 us; skew 500 ppm
Sep 24 13:42:52 dist-test-slave-tfm9 krb5kdc[27805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257372, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:52.551317 28048 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:42:52.551743 28048 webserver.cc:460] Webserver started at http://127.19.53.194:46817/ using document root <none> and password file <none>
I20260924 13:42:52.551977 28048 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:52.552021 28048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:52.552181 28048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:52.553225 28048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "1035f7c2953649c896b5959d72b6510c"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "806f915f95899f9cf94d5dd05a6a293d"
server_key_iv: "87d8fe4774b992861322b0239713d90e"
server_key_version: "encryptionkey@0"
I20260924 13:42:52.553563 28048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "1035f7c2953649c896b5959d72b6510c"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "806f915f95899f9cf94d5dd05a6a293d"
server_key_iv: "87d8fe4774b992861322b0239713d90e"
server_key_version: "encryptionkey@0"
I20260924 13:42:52.554874 28048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:52.555603 28063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:52.555783 28048 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:52.555871 28048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "1035f7c2953649c896b5959d72b6510c"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "806f915f95899f9cf94d5dd05a6a293d"
server_key_iv: "87d8fe4774b992861322b0239713d90e"
server_key_version: "encryptionkey@0"
I20260924 13:42:52.555949 28048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:52.569880 28048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:52.571746 28048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:52.571903 28048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:52.572132 28048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:52.572427 28048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:52.572463 28048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:52.572506 28048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:52.572533 28048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:52.579023 28048 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:33217
I20260924 13:42:52.579061 28177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:33217 every 8 connection(s)
I20260924 13:42:52.579438 28048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:42:52.583586 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 28048
I20260924 13:42:52.583685 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:42:52.583874 19671 external_mini_cluster.cc:1428] Setting key aa45bb75bfa3b5b6d36777fa70400317
Sep 24 13:42:52 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257372, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:52.589484 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:52.580269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:53353 (local address 127.19.53.254:38999)
0924 13:42:52.580332 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:52.580334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:52.580553 (+ 219us) server_negotiation.cc:407] Connection header received
0924 13:42:52.581071 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:52.581072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:52.581081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:52.581136 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:52.582153 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.582634 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.583949 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.584273 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.584340 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:52.586714 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:52.586715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:52.586716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:52.586730 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:52.588108 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:52.588536 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:52.588537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:52.588538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:52.588598 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:52.588847 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:52.588848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:52.588849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:52.588968 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:52.589045 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:52.589317 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 13:42:52.589394 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:42:52.589673 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:52.580237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:53353)
0924 13:42:52.580405 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:52.580451 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:52.580978 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:52.581195 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:52.581198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:52.581827 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:52.582069 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:52.582075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.582821 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.582822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:52.583846 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:52.583850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.584351 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.584352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:52.584481 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:52.584978 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:52.584988 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:52.586607 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:52.588209 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:52.588216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:52.588218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:52.588445 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:52.588666 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:52.588666 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:52.588667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:52.588779 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:52.589035 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:52.589042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:52.589181 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 13:42:52.589284 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 13:42:52.589447 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260924 13:42:52.590080 28178 heartbeater.cc:347] Connected to a master server at 127.19.53.254:38999
I20260924 13:42:52.590185 28178 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:52.590350 28178 heartbeater.cc:511] Master 127.19.53.254:38999 requested a full tablet report, sending...
I20260924 13:42:52.590698 27851 ts_manager.cc:194] Registered new tserver with Master: 1035f7c2953649c896b5959d72b6510c (127.19.53.194:33217)
I20260924 13:42:52.591114 27851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:53353
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:42:52.608357 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38999
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:52.687979 28183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:52.688169 28183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:52.688191 28183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:52.689280 28183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:52.689314 28183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:52.689339 28183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:52.690752 28183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:52.690937 28183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:52.691100 28183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:42:52.695876 28183 server_base.cc:1034] running on GCE node
W20260924 13:42:52.695858 28188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:52.695859 28189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:52.695837 28191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:52.696363 28183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:52.696640 28183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:52.697798 28183 hybrid_clock.cc:648] HybridClock initialized: now 1790257372697795 us; error 34 us; skew 500 ppm
Sep 24 13:42:52 dist-test-slave-tfm9 krb5kdc[27805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257372, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:52.700243 28183 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:42:52.700642 28183 webserver.cc:460] Webserver started at http://127.19.53.195:43601/ using document root <none> and password file <none>
I20260924 13:42:52.700876 28183 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:52.700954 28183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:52.701097 28183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:52.702221 28183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "2f15ff041c54443aa472a66416cfacde"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "820849666dcdc002f970449e32c88158"
server_key_iv: "91967a270318ee48e8fdff30a6421df6"
server_key_version: "encryptionkey@0"
I20260924 13:42:52.702692 28183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "2f15ff041c54443aa472a66416cfacde"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "820849666dcdc002f970449e32c88158"
server_key_iv: "91967a270318ee48e8fdff30a6421df6"
server_key_version: "encryptionkey@0"
I20260924 13:42:52.703967 28183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:52.704687 28198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:52.704880 28183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:52.704983 28183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "2f15ff041c54443aa472a66416cfacde"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "820849666dcdc002f970449e32c88158"
server_key_iv: "91967a270318ee48e8fdff30a6421df6"
server_key_version: "encryptionkey@0"
I20260924 13:42:52.705057 28183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:52.713464 28183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:52.715677 28183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:52.715830 28183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:52.716033 28183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:52.716342 28183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:52.716394 28183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:52.716434 28183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:52.716460 28183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:52.722178 28183 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:44555
I20260924 13:42:52.722237 28312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:44555 every 8 connection(s)
I20260924 13:42:52.722564 28183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:42:52.722779 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 28183
I20260924 13:42:52.722904 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:42:52.723109 19671 external_mini_cluster.cc:1428] Setting key a822634c47e7ea28d35a6eb418e2ab72
Sep 24 13:42:52 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257372, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:42:52.731356 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:52.723274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:42759 (local address 127.19.53.254:38999)
0924 13:42:52.723331 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:52.723334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:52.723614 (+ 280us) server_negotiation.cc:407] Connection header received
0924 13:42:52.724231 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:52.724231 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:52.724240 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:52.724284 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:52.725140 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.725635 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.726664 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.726843 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.726897 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:52.728864 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:52.728866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:52.728866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:52.728872 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:52.730293 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:52.730631 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:52.730631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:52.730632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:52.730679 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:52.730876 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:52.730877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:52.730877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:52.730973 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:52.731022 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:52.731240 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 13:42:52.731295 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:42:52.731559 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:52.723275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:42759)
0924 13:42:52.723480 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:52.723547 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:52.724145 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:52.724337 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:52.724341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:52.724877 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:52.725058 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:52.725063 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.725713 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.725714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:52.726595 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:52.726597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:52.726912 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:52.726913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:52.727009 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:52.727445 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:52.727452 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:52.728785 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:52.730359 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:52.730364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:52.730365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:52.730567 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:52.730734 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:52.730734 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:52.730734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:52.730825 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:52.731026 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:52.731029 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:52.731134 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 13:42:52.731212 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 13:42:52.731322 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260924 13:42:52.732043 28313 heartbeater.cc:347] Connected to a master server at 127.19.53.254:38999
I20260924 13:42:52.732117 28313 heartbeater.cc:464] Registering TS with master...
I20260924 13:42:52.732297 28313 heartbeater.cc:511] Master 127.19.53.254:38999 requested a full tablet report, sending...
I20260924 13:42:52.732697 27851 ts_manager.cc:194] Registered new tserver with Master: 2f15ff041c54443aa472a66416cfacde (127.19.53.195:44555)
I20260924 13:42:52.733047 27851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:42759
I20260924 13:42:52.734982 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab.
Entry for principal oryx/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab.
WARNING: no policy specified for oryx/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab.
Entry for principal oryx/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab.
WARNING: no policy specified for oryx/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab.
Entry for principal oryx/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab.
I20260924 13:42:52.808399 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 27913
I20260924 13:42:52.813660 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 28048
I20260924 13:42:52.817905 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 28183
I20260924 13:42:52.822124 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 27818
I20260924 13:42:52.827369 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:38999
--webserver_interface=127.19.53.254
--webserver_port=35777
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:38999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:52.906986 28322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:52.907207 28322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:52.907243 28322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:52.908394 28322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:52.908463 28322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:52.908488 28322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:52.908517 28322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:52.908548 28322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:52.910147 28322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:38999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:38999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=35777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:52.910413 28322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:52.910640 28322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:52.915489 28327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:52.915534 28328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:52.915741 28322 server_base.cc:1034] running on GCE node
W20260924 13:42:52.915489 28331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:52.916069 28322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:52.916348 28322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:52.917486 28322 hybrid_clock.cc:648] HybridClock initialized: now 1790257372917477 us; error 30 us; skew 500 ppm
Sep 24 13:42:52 dist-test-slave-tfm9 krb5kdc[27805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257372, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:52.920131 28322 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:42:52.920583 28322 webserver.cc:460] Webserver started at http://127.19.53.254:35777/ using document root <none> and password file <none>
I20260924 13:42:52.920804 28322 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:52.920859 28322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:52.922227 28322 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:52.922871 28338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:52.923048 28322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:52.923149 28322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "d5bd2fdade1558d9a3ed2db2c0c9cbc1"
server_key_iv: "3cfd82995c9d51470c403c46c1555e24"
server_key_version: "encryptionkey@0"
I20260924 13:42:52.923475 28322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:52.947633 28322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:52.949661 28322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:52.949839 28322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:52.953897 28322 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:38999
I20260924 13:42:52.954005 28390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:38999 every 8 connection(s)
I20260924 13:42:52.954421 28322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:52.955862 28391 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:42:52.956552 28391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b: Bootstrap starting.
I20260924 13:42:52.962224 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 28322
I20260924 13:42:52.962546 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:35809
--local_ip_for_outbound_sockets=127.19.53.193
--tserver_master_addrs=127.19.53.254:38999
--webserver_port=40349
--webserver_interface=127.19.53.193
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.53.193
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:42:52.964766 28391 log.cc:824] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:52.966432 28391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:42:52.966718 28391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b: Bootstrap complete.
I20260924 13:42:52.969007 28391 raft_consensus.cc:348] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38999 } }
I20260924 13:42:52.969251 28391 raft_consensus.cc:729] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77c3edcd81c4f0c8f82783bc6d6e56b, State: Initialized, Role: FOLLOWER
I20260924 13:42:52.969432 28391 consensus_queue.cc:260] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38999 } }
I20260924 13:42:52.969512 28391 raft_consensus.cc:388] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:52.969566 28391 raft_consensus.cc:482] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:52.969623 28391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:42:52.971009 28391 raft_consensus.cc:504] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38999 } }
I20260924 13:42:52.971118 28391 leader_election.cc:302] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a77c3edcd81c4f0c8f82783bc6d6e56b; no voters:
I20260924 13:42:52.971269 28391 leader_election.cc:288] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 13:42:52.971307 28395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:42:52.971434 28395 raft_consensus.cc:686] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [term 2 LEADER]: Becoming Leader. State: Replica: a77c3edcd81c4f0c8f82783bc6d6e56b, State: Running, Role: LEADER
I20260924 13:42:52.971527 28395 consensus_queue.cc:237] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38999 } }
I20260924 13:42:52.971531 28391 sys_catalog.cc:564] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:52.971788 28397 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a77c3edcd81c4f0c8f82783bc6d6e56b. Latest consensus state: current_term: 2 leader_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38999 } } }
I20260924 13:42:52.971853 28397 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:52.971995 28396 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c3edcd81c4f0c8f82783bc6d6e56b" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38999 } } }
I20260924 13:42:52.972041 28396 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:52.972039 28399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:52.972581 28399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:52.972714 28399 catalog_manager.cc:1259] Loaded cluster ID: 448e043a033f4b0a87abb1beebc02b63
I20260924 13:42:52.972741 28399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:52.973928 28399 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:52.974097 28399 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a77c3edcd81c4f0c8f82783bc6d6e56b: Loaded TSK: 0
I20260924 13:42:52.974311 28399 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:42:53.044337 28393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:53.044565 28393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:53.044603 28393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:53.045758 28393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:53.045825 28393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:53.045871 28393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:42:53.047434 28393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.193.keytab
--principal=oryx/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:35809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=40349
--tserver_master_addrs=127.19.53.254:38999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:53.047696 28393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:53.047931 28393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:53.053818 28419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:53.053798 28421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:53.053798 28418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:53.054052 28393 server_base.cc:1034] running on GCE node
I20260924 13:42:53.054284 28393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:53.054540 28393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:53.055694 28393 hybrid_clock.cc:648] HybridClock initialized: now 1790257373055674 us; error 38 us; skew 500 ppm
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257373, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:53.058529 28393 init.cc:377] Logged in from keytab as oryx/127.19.53.193@KRBTEST.COM (short username oryx)
I20260924 13:42:53.059053 28393 webserver.cc:460] Webserver started at http://127.19.53.193:40349/ using document root <none> and password file <none>
I20260924 13:42:53.059338 28393 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:53.059422 28393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:53.061192 28393 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:53.061858 28428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:53.062073 28393 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:53.062176 28393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "37389a2770b74236b53312ef250af8ea"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "243a6b99ac423d30d624f73f45f0e198"
server_key_iv: "4b7d8c74b1b05c36ccfb1d50042621ad"
server_key_version: "encryptionkey@0"
I20260924 13:42:53.062515 28393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:53.071323 28393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:53.073700 28393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:53.073855 28393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:53.074098 28393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:53.074388 28393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:53.074442 28393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:53.074486 28393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:53.074513 28393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:53.080308 28393 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:35809
I20260924 13:42:53.080354 28542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:35809 every 8 connection(s)
I20260924 13:42:53.080719 28393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:42:53.087405 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 28393
I20260924 13:42:53.087702 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:33217
--local_ip_for_outbound_sockets=127.19.53.194
--tserver_master_addrs=127.19.53.254:38999
--webserver_port=46817
--webserver_interface=127.19.53.194
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.53.194
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:53.089980 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:53.081405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:35595)
0924 13:42:53.081615 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:53.081661 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:53.082312 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:53.082744 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:53.082749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:53.083469 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:53.083723 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.083728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.084811 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.084812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.086067 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.086070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.086629 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.086630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.086765 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.087236 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:53.087245 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:53.089739 (+ 2494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":122,"threads_started":1}
W20260924 13:42:53.090198 28543 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:38999 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.53.254:38999: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
W20260924 13:42:53.090407 28546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:53.081554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:35595 (local address 127.19.53.254:38999)
0924 13:42:53.081738 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:53.081746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:53.081772 (+ 26us) server_negotiation.cc:407] Connection header received
0924 13:42:53.082507 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:53.082512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:53.082554 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:53.082666 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:53.083829 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.084714 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.086144 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.086546 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.086623 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.089966 (+ 3343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:35595: BlockingRecv error: recv got EOF from 127.19.53.193:35595 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:53.095571 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:53.090375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:49811)
0924 13:42:53.090541 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:53.090550 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:53.090585 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:53.090855 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:53.090856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:53.090989 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:53.091041 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.091043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.091626 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.091628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.092426 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.092428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.092982 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.092983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.093057 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.093486 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:53.093491 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:53.095512 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
W20260924 13:42:53.096113 28546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:53.090633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:49811 (local address 127.19.53.254:38999)
0924 13:42:53.090769 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:53.090771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:53.090777 (+ 6us) server_negotiation.cc:407] Connection header received
0924 13:42:53.090795 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:53.090795 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:53.090803 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:53.090849 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:53.091120 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.091549 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.092522 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.092885 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.092956 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.095800 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:49811: BlockingRecv error: recv got EOF from 127.19.53.193:49811 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:53.100960 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:53.095803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:40983)
0924 13:42:53.095871 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:53.095880 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:53.095911 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:53.096054 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:53.096054 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:53.096182 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:53.096209 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.096210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.097065 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.097066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.097953 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.097956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.098437 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.098438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.098503 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.098926 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:53.098931 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:53.100879 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 13:42:53.101150 28543 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:42:53.101240 28549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:53.095871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:40983 (local address 127.19.53.254:38999)
0924 13:42:53.095920 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:53.095922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:53.095931 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:53.095954 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:53.095955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:53.095965 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:53.096046 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:53.096286 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.096983 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.098032 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.098346 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.098418 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.100983 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:40983: BlockingRecv error: recv got EOF from 127.19.53.193:40983 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 13:42:53.171379 28547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:53.171630 28547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:53.171671 28547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:53.173094 28547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:53.173151 28547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:53.173197 28547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:42:53.174598 28547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.194.keytab
--principal=oryx/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:33217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=46817
--tserver_master_addrs=127.19.53.254:38999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:53.174844 28547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:53.175081 28547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:53.179875 28554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:53.179948 28555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:53.179885 28557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:53.180120 28547 server_base.cc:1034] running on GCE node
I20260924 13:42:53.180321 28547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:53.180617 28547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:53.181764 28547 hybrid_clock.cc:648] HybridClock initialized: now 1790257373181754 us; error 26 us; skew 500 ppm
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257373, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:53.184026 28547 init.cc:377] Logged in from keytab as oryx/127.19.53.194@KRBTEST.COM (short username oryx)
I20260924 13:42:53.184475 28547 webserver.cc:460] Webserver started at http://127.19.53.194:46817/ using document root <none> and password file <none>
I20260924 13:42:53.184685 28547 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:53.184756 28547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:53.186090 28547 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:53.186635 28564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:53.186815 28547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:53.186900 28547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "1035f7c2953649c896b5959d72b6510c"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "806f915f95899f9cf94d5dd05a6a293d"
server_key_iv: "87d8fe4774b992861322b0239713d90e"
server_key_version: "encryptionkey@0"
I20260924 13:42:53.187211 28547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:53.214411 28547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:53.216805 28547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:53.216971 28547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:53.217200 28547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:53.217514 28547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:53.217566 28547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:53.217604 28547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:53.217633 28547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:53.223295 28547 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:33217
I20260924 13:42:53.223366 28678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:33217 every 8 connection(s)
I20260924 13:42:53.223717 28547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:42:53.226281 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 28547
I20260924 13:42:53.226627 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:44555
--local_ip_for_outbound_sockets=127.19.53.195
--tserver_master_addrs=127.19.53.254:38999
--webserver_port=43601
--webserver_interface=127.19.53.195
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.53.195
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 13:42:53.232159 28549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:53.224507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:32835 (local address 127.19.53.254:38999)
0924 13:42:53.224551 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:53.224570 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:53.224775 (+ 205us) server_negotiation.cc:407] Connection header received
0924 13:42:53.225317 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:53.225318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:53.225334 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:53.225398 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:53.226616 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.227248 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.228554 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.228871 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.228949 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.232091 (+ 3142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:32835: BlockingRecv error: recv got EOF from 127.19.53.194:32835 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:53.232252 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:53.224437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:32835)
0924 13:42:53.224624 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:53.224668 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:53.225199 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:53.225466 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:53.225470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:53.226163 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:53.226502 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.226508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.227339 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.227340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.228468 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.228470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.228959 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.228961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.229102 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.229569 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:53.229579 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:53.232002 (+ 2423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
W20260924 13:42:53.232474 28679 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:38999 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.53.254:38999: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:53.236806 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:53.232708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:57923)
0924 13:42:53.232741 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:53.232751 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:53.232812 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:53.232940 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:53.232941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:53.233064 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:53.233101 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.233103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.233724 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.233725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.234269 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.234270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.234627 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.234628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.234689 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.235134 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:53.235138 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:53.236732 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 13:42:53.236857 28549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:53.232763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:57923 (local address 127.19.53.254:38999)
0924 13:42:53.232821 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:53.232823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:53.232834 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:42:53.232868 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:53.232869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:53.232879 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:53.232920 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:53.233173 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.233633 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.234359 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.234544 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.234627 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.236796 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:57923: BlockingRecv error: recv got EOF from 127.19.53.194:57923 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:53.240954 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:53.237056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:36809)
0924 13:42:53.237091 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:53.237100 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:53.237140 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:53.237271 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:53.237271 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:53.237389 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:53.237411 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.237412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.238017 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.238018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.238478 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.238479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.238841 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.238841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.238891 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.239283 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:53.239286 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:53.240891 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 13:42:53.241010 28549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:53.237131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:36809 (local address 127.19.53.254:38999)
0924 13:42:53.237185 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:53.237187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:53.237195 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:53.237214 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:53.237214 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:53.237223 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:53.237269 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:53.237481 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.237960 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.238555 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.238759 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.238839 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.240954 (+ 2115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:36809: BlockingRecv error: recv got EOF from 127.19.53.194:36809 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 13:42:53.241154 28679 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:42:53.315451 28682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:53.315711 28682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:53.315757 28682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:53.317021 28682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:53.317098 28682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:53.317145 28682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:42:53.318627 28682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/krb5kdc/oryx_127.19.53.195.keytab
--principal=oryx/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:44555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=43601
--tserver_master_addrs=127.19.53.254:38999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:53.318898 28682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:53.319135 28682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:53.324137 28688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:53.324142 28690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:53.324180 28687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:53.324456 28682 server_base.cc:1034] running on GCE node
I20260924 13:42:53.324671 28682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:53.324867 28682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:53.325984 28682 hybrid_clock.cc:648] HybridClock initialized: now 1790257373325968 us; error 30 us; skew 500 ppm
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257373, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:53.328567 28682 init.cc:377] Logged in from keytab as oryx/127.19.53.195@KRBTEST.COM (short username oryx)
I20260924 13:42:53.329057 28682 webserver.cc:460] Webserver started at http://127.19.53.195:43601/ using document root <none> and password file <none>
I20260924 13:42:53.329285 28682 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:53.329370 28682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:53.330790 28682 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:53.331451 28697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:53.331642 28682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:53.331736 28682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "2f15ff041c54443aa472a66416cfacde"
format_stamp: "Formatted at 2026-09-24 13:42:52 on dist-test-slave-tfm9"
server_key: "820849666dcdc002f970449e32c88158"
server_key_iv: "91967a270318ee48e8fdff30a6421df6"
server_key_version: "encryptionkey@0"
I20260924 13:42:53.332084 28682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:53.348033 28682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:53.350564 28682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:53.350741 28682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:53.350996 28682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:42:53.351320 28682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:42:53.351356 28682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:53.351382 28682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:42:53.351433 28682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:53.357776 28682 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:44555
I20260924 13:42:53.357831 28811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:44555 every 8 connection(s)
I20260924 13:42:53.358198 28682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:42:53.362099 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 28682
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 13:42:53.367285 28549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:53.359001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:47043 (local address 127.19.53.254:38999)
0924 13:42:53.359044 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:53.359046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:53.359449 (+ 403us) server_negotiation.cc:407] Connection header received
0924 13:42:53.360069 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:53.360071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:53.360087 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:53.360163 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:53.361310 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.362073 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.363445 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.363661 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.363746 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.367236 (+ 3490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:47043: BlockingRecv error: recv got EOF from 127.19.53.195:47043 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 13:42:53.367383 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:53.358956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:47043)
0924 13:42:53.359236 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:53.359278 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:53.359894 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:53.360230 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:53.360234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:53.361009 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:53.361207 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.361212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.362213 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.362214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.363339 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.363342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.363756 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.363757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.363890 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.364408 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:53.364418 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:53.367163 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":89,"threads_started":1}
W20260924 13:42:53.367810 28812 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:38999 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.53.254:38999: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
I20260924 13:42:53.370051 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:53.362615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.19.53.254:38999)
0924 13:42:53.362664 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:53.362667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:53.362775 (+ 108us) server_negotiation.cc:407] Connection header received
0924 13:42:53.362821 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:53.362822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:53.362835 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:53.362887 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:53.363470 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.363930 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.364739 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.365046 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.365118 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.366570 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:53.366575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:53.366575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:53.366586 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:53.368619 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:53.368937 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:53.368940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:53.368941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:53.369056 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:53.369340 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:53.369341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:53.369341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:53.369554 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:53.369654 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:53.369825 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 13:42:53.369905 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:53.373039 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:53.368056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:59709)
0924 13:42:53.368158 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:53.368172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:53.368210 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:53.368335 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:53.368336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:53.368445 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:53.368483 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.368484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.369257 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.369259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.370143 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.370145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.370728 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.370728 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.370806 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.371190 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:53.371195 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:53.372956 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86}
W20260924 13:42:53.373087 28549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:53.368117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:59709 (local address 127.19.53.254:38999)
0924 13:42:53.368153 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:53.368155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:53.368201 (+ 46us) server_negotiation.cc:407] Connection header received
0924 13:42:53.368264 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:53.368265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:53.368274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:53.368333 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:53.368550 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.369168 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.370260 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.370583 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.370643 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.373024 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:59709: BlockingRecv error: recv got EOF from 127.19.53.195:59709 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:53 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:53.377085 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:53.373263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:36137)
0924 13:42:53.373443 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:53.373453 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:53.373502 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:53.373606 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:53.373607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:53.373712 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:53.373737 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.373738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.374324 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.374325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.374833 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:53.374834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.375133 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.375134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:53.375180 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.375473 (+ 293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:53.375477 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:53.377029 (+ 1552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
W20260924 13:42:53.377138 28549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:53.373384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:36137 (local address 127.19.53.254:38999)
0924 13:42:53.373454 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:53.373456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:53.373493 (+ 37us) server_negotiation.cc:407] Connection header received
0924 13:42:53.373557 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:53.373557 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:53.373564 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:53.373602 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:53.373791 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.374260 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.374899 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:53.375072 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:53.375119 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:53.377103 (+ 1984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:36137: BlockingRecv error: recv got EOF from 127.19.53.195:36137 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 13:42:53.377275 28812 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 13:42:54 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:54 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:54.106592 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:54.101681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:40517)
0924 13:42:54.101840 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:54.101852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:54.101959 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:54.102138 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:54.102138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:54.102294 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:54.102343 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:54.102345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:54.102995 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:54.102996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:54.103519 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:54.103521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:54.103869 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:54.103870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:54.103906 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:54.104390 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:54.104394 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:54.106481 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
W20260924 13:42:54.106634 28818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:54.101788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:40517 (local address 127.19.53.254:38999)
0924 13:42:54.101904 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:54.101922 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:54.101932 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:42:54.102045 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:54.102046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:54.102057 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:54.102120 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:54.102404 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:54.102922 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:54.103589 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:54.103789 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:54.103861 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:54.106561 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:40517: BlockingRecv error: recv got EOF from 127.19.53.193:40517 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
Sep 24 13:42:54 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:54 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:54.246456 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:54.241659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:37697)
0924 13:42:54.241826 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:54.241837 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:54.241901 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:54.242050 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:54.242051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:54.242231 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:54.242284 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:54.242286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:54.242968 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:54.242969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:54.243538 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:54.243540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:54.243861 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:54.243861 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:54.243901 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:54.244357 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:54.244361 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:54.246390 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
W20260924 13:42:54.246592 28818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:54.241740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:37697 (local address 127.19.53.254:38999)
0924 13:42:54.241790 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:54.241792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:54.241897 (+ 105us) server_negotiation.cc:407] Connection header received
0924 13:42:54.241969 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:54.241970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:54.241980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:54.242035 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:54.242357 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:54.242847 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:54.243615 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:54.243793 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:54.243843 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:54.246513 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:37697: BlockingRecv error: recv got EOF from 127.19.53.194:37697 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 13:42:54 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:54 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:54.382153 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:54.377684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:38579)
0924 13:42:54.377812 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:54.377824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:54.377888 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:54.378084 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:54.378085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:54.378236 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:54.378282 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:54.378283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:54.378970 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:54.378971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:54.379547 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:54.379548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:54.379864 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:54.379865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:54.379917 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:54.380368 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:54.380372 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:54.382080 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
W20260924 13:42:54.382218 28818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:54.377762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:38579 (local address 127.19.53.254:38999)
0924 13:42:54.377848 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:54.377850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:54.377872 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:42:54.377973 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:54.377974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:54.377997 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:54.378065 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:54.378387 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:54.378896 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:54.379619 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:54.379799 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:54.379848 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:54.382170 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:38579: BlockingRecv error: recv got EOF from 127.19.53.195:38579 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 13:42:55 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:55 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:55.111740 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:55.107069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:48281)
0924 13:42:55.107245 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:55.107255 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:55.107312 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:55.107465 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:55.107466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:55.107635 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:55.107688 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:55.107689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:55.108335 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:55.108336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:55.108864 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:55.108866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:55.109192 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:55.109192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:55.109235 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:55.109684 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:55.109689 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:55.111613 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
W20260924 13:42:55.111832 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:55.107165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:48281 (local address 127.19.53.254:38999)
0924 13:42:55.107294 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:55.107296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:55.107305 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:55.107372 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:55.107372 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:55.107383 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:55.107447 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:55.107751 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:55.108263 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:55.108943 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:55.109127 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:55.109176 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:55.111747 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:48281: BlockingRecv error: recv got EOF from 127.19.53.193:48281 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
Sep 24 13:42:55 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:55 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:55.251777 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:55.246897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:58941)
0924 13:42:55.247072 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:55.247086 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:55.247179 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:55.247378 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:55.247379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:55.247557 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:55.247604 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:55.247606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:55.248301 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:55.248302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:55.248837 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:55.248838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:55.249201 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:55.249202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:55.249265 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:55.249753 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:55.249762 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:55.251719 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
W20260924 13:42:55.251945 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:55.246989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:58941 (local address 127.19.53.254:38999)
0924 13:42:55.247076 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:55.247078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:55.247159 (+ 81us) server_negotiation.cc:407] Connection header received
0924 13:42:55.247267 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:55.247268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:55.247279 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:55.247372 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:55.247684 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:55.248217 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:55.248920 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:55.249115 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:55.249198 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:55.251820 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:58941: BlockingRecv error: recv got EOF from 127.19.53.194:58941 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 13:42:55 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:55 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:55.387177 28824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:55.382591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:36869)
0924 13:42:55.382753 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:55.382767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:55.382837 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:55.383000 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:55.383001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:55.383171 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:55.383218 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:55.383220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:55.383933 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:55.383934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:55.384471 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:55.384472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:55.384840 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:55.384841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:55.384906 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:55.385322 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:55.385327 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:55.387081 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
W20260924 13:42:55.387217 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:55.382702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:36869 (local address 127.19.53.254:38999)
0924 13:42:55.382770 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:55.382772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:55.382826 (+ 54us) server_negotiation.cc:407] Connection header received
0924 13:42:55.382908 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:55.382909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:55.382923 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:55.382996 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:55.383299 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:55.383833 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:55.384556 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:55.384754 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:55.384838 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:55.387157 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:36869: BlockingRecv error: recv got EOF from 127.19.53.195:36869 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 13:42:56 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:56 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:56.117112 28825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:56.112300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:56281)
0924 13:42:56.112450 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:56.112463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:56.112530 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:56.112673 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:56.112673 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:56.112888 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:56.112951 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:56.112954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:56.113842 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:56.113843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:56.114475 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:56.114477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:56.114810 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:56.114810 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:56.114849 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:56.115291 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:56.115295 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:56.117054 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
W20260924 13:42:56.117249 28826 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:56.112362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:56281 (local address 127.19.53.254:38999)
0924 13:42:56.112502 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:56.112504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:56.112514 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:42:56.112586 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:56.112586 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:56.112597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:56.112654 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:56.113042 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:56.113774 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:56.114548 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:56.114745 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:56.114818 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:56.117160 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:56281: BlockingRecv error: recv got EOF from 127.19.53.193:56281 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
Sep 24 13:42:56 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:56 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:56.257160 28827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:56.252371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:55477)
0924 13:42:56.252601 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:56.252611 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:56.252687 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:56.252814 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:56.252815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:56.253007 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:56.253052 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:56.253054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:56.253717 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:56.253718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:56.254255 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:56.254256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:56.254627 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:56.254628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:56.254691 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:56.255181 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:56.255186 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:56.257100 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":121,"threads_started":1}
W20260924 13:42:56.257294 28826 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:56.252448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:55477 (local address 127.19.53.254:38999)
0924 13:42:56.252525 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:56.252527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:56.252677 (+ 150us) server_negotiation.cc:407] Connection header received
0924 13:42:56.252753 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:56.252754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:56.252765 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:56.252810 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:56.253137 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:56.253633 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:56.254341 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:56.254544 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:56.254626 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:56.257209 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:55477: BlockingRecv error: recv got EOF from 127.19.53.194:55477 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 13:42:56 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:56 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:56.392467 28828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:56.387735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:37379)
0924 13:42:56.387872 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:56.387882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:56.388004 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:56.388192 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:56.388193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:56.388395 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:56.388458 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:56.388460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:56.389224 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:56.389225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:56.389760 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:56.389762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:56.390192 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:56.390193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:56.390237 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:56.390684 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:56.390688 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:56.392406 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
W20260924 13:42:56.392560 28826 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:56.387812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:37379 (local address 127.19.53.254:38999)
0924 13:42:56.387871 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:56.387873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:56.387987 (+ 114us) server_negotiation.cc:407] Connection header received
0924 13:42:56.388079 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:56.388080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:56.388096 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:56.388162 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:56.388559 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:56.389129 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:56.389854 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:56.390116 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:56.390190 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:56.392491 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:37379: BlockingRecv error: recv got EOF from 127.19.53.195:37379 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 13:42:57 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:57 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:57.123610 28829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:57.117656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:54159)
0924 13:42:57.117777 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:57.117791 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:57.117880 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:57.118124 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:57.118126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:57.118334 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:57.118403 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:57.118406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:57.119142 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:57.119144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:57.120082 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:57.120085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:57.120603 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:57.120605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:57.120678 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:57.121384 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:57.121390 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:57.123509 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
W20260924 13:42:57.123672 28830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:57.117697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:54159 (local address 127.19.53.254:38999)
0924 13:42:57.117808 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:57.117810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:57.117855 (+ 45us) server_negotiation.cc:407] Connection header received
0924 13:42:57.117983 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:57.117984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:57.118019 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:57.118103 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:57.118482 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:57.119048 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:57.120175 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:57.120497 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:57.120557 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:57.123596 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:54159: BlockingRecv error: recv got EOF from 127.19.53.193:54159 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":48,"threads_started":1}
Sep 24 13:42:57 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:57 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:57.270133 28831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:57.257706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:34795)
0924 13:42:57.258007 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:57.258017 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:57.258075 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:57.258227 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:57.258228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:57.258413 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:57.258460 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:57.258462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:57.259142 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:57.259143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:57.259689 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:57.259691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:57.260070 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:57.260070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:57.260123 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:57.268104 (+ 7981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:57.268110 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:57.270034 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":233,"threads_started":1}
W20260924 13:42:57.270188 28830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:57.257896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:34795 (local address 127.19.53.254:38999)
0924 13:42:57.258002 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:57.258004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:57.258099 (+ 95us) server_negotiation.cc:407] Connection header received
0924 13:42:57.258132 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:57.258133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:57.258147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:57.258212 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:57.258530 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:57.259075 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:57.259771 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:57.260000 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:57.260060 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:57.270121 (+ 10061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:34795: BlockingRecv error: recv got EOF from 127.19.53.194:34795 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 13:42:57 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:57 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:57.397717 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:57.392935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:47275)
0924 13:42:57.393091 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:57.393101 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:57.393155 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:57.393283 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:57.393283 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:57.393447 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:57.393498 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:57.393499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:57.394347 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:57.394348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:57.394998 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:57.395001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:57.395320 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:57.395320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:57.395357 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:57.395874 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:57.395878 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:57.397620 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
W20260924 13:42:57.397753 28830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:57.393002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:47275 (local address 127.19.53.254:38999)
0924 13:42:57.393076 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:57.393078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:57.393154 (+ 76us) server_negotiation.cc:407] Connection header received
0924 13:42:57.393199 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:57.393199 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:57.393211 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:57.393266 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:57.393556 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:57.394223 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:57.395066 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:57.395256 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:57.395310 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:57.397693 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:47275: BlockingRecv error: recv got EOF from 127.19.53.195:47275 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 13:42:58 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:58 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:58.129020 28833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:58.124169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:56355)
0924 13:42:58.124332 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:58.124343 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:58.124400 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:58.124526 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:58.124526 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:58.124702 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:58.124753 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:58.124755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:58.125541 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:58.125542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:58.126143 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:58.126145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:58.126481 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:58.126481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:58.126562 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:58.127082 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:58.127087 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:58.128921 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
W20260924 13:42:58.129056 28834 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:58.124265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:56355 (local address 127.19.53.254:38999)
0924 13:42:58.124394 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:58.124396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:58.124406 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:42:58.124453 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:58.124454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:58.124465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:58.124520 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:58.124817 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:58.125419 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:58.126218 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:58.126412 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:58.126467 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:58.128991 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:56355: BlockingRecv error: recv got EOF from 127.19.53.193:56355 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
Sep 24 13:42:58 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:58 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:58.275506 28835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:58.270614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:58139)
0924 13:42:58.270794 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:58.270808 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:58.270889 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:58.271080 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:58.271081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:58.271251 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:58.271308 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:58.271309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:58.271952 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:58.271953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:58.272615 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:58.272617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:58.272990 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:58.272991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:58.273041 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:58.273595 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:58.273601 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:58.275446 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
W20260924 13:42:58.275687 28834 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:58.270762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:58139 (local address 127.19.53.254:38999)
0924 13:42:58.270824 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:58.270825 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:58.270869 (+ 44us) server_negotiation.cc:407] Connection header received
0924 13:42:58.270988 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:58.270989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:58.271001 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:58.271057 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:58.271383 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:58.271877 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:58.272696 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:58.272904 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:58.272963 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:58.275601 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:58139: BlockingRecv error: recv got EOF from 127.19.53.194:58139 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 13:42:58 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:58 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:58.402942 28836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:58.398216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:33187)
0924 13:42:58.398384 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:58.398394 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:58.398477 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:58.398643 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:58.398644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:58.398832 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:58.398879 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:58.398881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:58.399572 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:58.399573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:58.400109 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:58.400110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:58.400500 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:58.400500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:58.400556 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:58.401017 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:58.401022 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:58.402855 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
W20260924 13:42:58.402959 28834 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:58.398272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:33187 (local address 127.19.53.254:38999)
0924 13:42:58.398345 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:58.398347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:58.398458 (+ 111us) server_negotiation.cc:407] Connection header received
0924 13:42:58.398541 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:58.398542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:58.398557 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:58.398623 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:58.398955 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:58.399484 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:58.400207 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:58.400425 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:58.400502 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:58.402908 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:33187: BlockingRecv error: recv got EOF from 127.19.53.195:33187 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 13:42:59 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:59 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:59.134902 28837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:59.129621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:43345)
0924 13:42:59.129763 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:59.129773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:59.129824 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:59.130033 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:59.130033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:59.130188 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:59.130257 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:59.130259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:59.130945 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:59.130945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:59.131508 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:59.131510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:59.132159 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:59.132161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:59.132230 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:59.132778 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:59.132784 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:59.134839 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
W20260924 13:42:59.135087 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:59.129672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:43345 (local address 127.19.53.254:38999)
0924 13:42:59.129801 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:59.129803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:59.129813 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:42:59.129889 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:59.129889 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:59.129900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:59.130010 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:59.130384 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:59.130879 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:59.131677 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:59.132042 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:59.132149 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:59.135016 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:43345: BlockingRecv error: recv got EOF from 127.19.53.193:43345 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
Sep 24 13:42:59 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:59 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:59.280754 28839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:59.275972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:41545)
0924 13:42:59.276155 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:59.276166 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:59.276253 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:59.276446 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:59.276447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:59.276624 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:59.276671 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:59.276673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:59.277351 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:59.277352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:59.277889 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:59.277890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:59.278285 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:59.278286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:59.278350 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:59.278845 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:59.278850 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:59.280696 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
W20260924 13:42:59.280882 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:59.276076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:41545 (local address 127.19.53.254:38999)
0924 13:42:59.276149 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:59.276151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:59.276235 (+ 84us) server_negotiation.cc:407] Connection header received
0924 13:42:59.276331 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:59.276332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:59.276346 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:59.276417 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:59.276752 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:59.277265 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:59.278002 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:59.278199 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:59.278284 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:59.280800 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:41545: BlockingRecv error: recv got EOF from 127.19.53.194:41545 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 13:42:59 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:42:59 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:42:59.407823 28840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:59.403388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:49651)
0924 13:42:59.403510 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:59.403521 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:59.403585 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:59.403711 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:59.403711 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:59.403868 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:59.403917 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:59.403918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:59.404552 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:59.404552 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:59.405072 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:59.405074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:59.405400 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:59.405401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:59.405439 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:59.405980 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:59.405985 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:59.407763 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
W20260924 13:42:59.407953 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:59.403458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:49651 (local address 127.19.53.254:38999)
0924 13:42:59.403511 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:59.403512 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:59.403570 (+ 58us) server_negotiation.cc:407] Connection header received
0924 13:42:59.403642 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:59.403642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:59.403653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:59.403695 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:59.403979 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:59.404477 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:59.405146 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:59.405328 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:59.405384 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:59.407872 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:49651: BlockingRecv error: recv got EOF from 127.19.53.195:49651 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 13:43:00 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:00 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:00.140339 28841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:00.135477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:36515)
0924 13:43:00.135633 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:00.135663 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:00.135754 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:00.135958 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:00.135959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:00.136188 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:00.136251 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:00.136253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:00.136930 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:00.136931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:00.137531 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:00.137532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:00.137883 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:00.137883 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:00.137969 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:00.138455 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:00.138460 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:00.140279 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
W20260924 13:43:00.140497 28842 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:00.135581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:36515 (local address 127.19.53.254:38999)
0924 13:43:00.135717 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:00.135719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:00.135727 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:43:00.135854 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:00.135855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:00.135870 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:00.135936 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:00.136326 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:00.136866 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:00.137600 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:00.137798 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:00.137872 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:00.140401 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:36515: BlockingRecv error: recv got EOF from 127.19.53.193:36515 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
Sep 24 13:43:00 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:00 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:00.286155 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:00.281294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:36877)
0924 13:43:00.281431 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:00.281448 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:00.281505 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:00.281637 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:00.281638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:00.281809 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:00.281860 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:00.281862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:00.282563 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:00.282564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:00.283114 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:00.283116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:00.283460 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:00.283461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:00.283505 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:00.283953 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:00.283957 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:00.286095 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
W20260924 13:43:00.286247 28842 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:00.281355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:36877 (local address 127.19.53.254:38999)
0924 13:43:00.281449 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:00.281451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:00.281508 (+ 57us) server_negotiation.cc:407] Connection header received
0924 13:43:00.281555 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:00.281556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:00.281566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:00.281622 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:00.281963 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:00.282490 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:00.283187 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:00.283369 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:00.283444 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:00.286182 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:36877: BlockingRecv error: recv got EOF from 127.19.53.194:36877 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 13:43:00 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:00 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:00.413151 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:00.408386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:55827)
0924 13:43:00.408579 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:00.408590 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:00.408677 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:00.408869 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:00.408870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:00.409046 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:00.409092 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:00.409094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:00.409786 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:00.409787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:00.410379 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:00.410381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:00.410764 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:00.410765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:00.410823 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:00.411289 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:00.411315 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:00.413055 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":109,"threads_started":1}
W20260924 13:43:00.413189 28842 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:00.408485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:55827 (local address 127.19.53.254:38999)
0924 13:43:00.408572 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:00.408574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:00.408657 (+ 83us) server_negotiation.cc:407] Connection header received
0924 13:43:00.408761 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:00.408762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:00.408775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:00.408848 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:00.409189 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:00.409700 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:00.410467 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:00.410674 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:00.410763 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:00.413129 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:55827: BlockingRecv error: recv got EOF from 127.19.53.195:55827 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 13:43:01 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:01 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:01.145406 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:01.140834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:60845)
0924 13:43:01.141016 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:01.141028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:01.141092 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:01.141231 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:01.141231 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:01.141396 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:01.141446 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:01.141448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:01.142140 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:01.142141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:01.142658 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:01.142659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:01.142973 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:01.142973 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:01.143013 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:01.143541 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:01.143545 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:01.145329 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":93,"threads_started":1}
W20260924 13:43:01.145509 28846 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:01.140964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:60845 (local address 127.19.53.254:38999)
0924 13:43:01.141100 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:01.141101 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:01.141111 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:43:01.141147 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:01.141147 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:01.141157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:01.141206 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:01.141514 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:01.142074 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:01.142726 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:01.142912 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:01.142960 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:01.145419 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:60845: BlockingRecv error: recv got EOF from 127.19.53.193:60845 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
Sep 24 13:43:01 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:01 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:01.291904 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:01.286672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:49261)
0924 13:43:01.286862 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:01.286886 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:01.286969 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:01.287113 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:01.287113 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:01.287294 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:01.287345 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:01.287346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:01.288176 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:01.288177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:01.288708 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:01.288710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:01.289088 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:01.289089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:01.289145 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:01.289724 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:01.289730 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:01.291835 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
W20260924 13:43:01.291975 28846 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:01.286763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:49261 (local address 127.19.53.254:38999)
0924 13:43:01.286836 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:01.286839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:01.286948 (+ 109us) server_negotiation.cc:407] Connection header received
0924 13:43:01.287044 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:01.287045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:01.287059 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:01.287106 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:01.287426 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:01.288088 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:01.288800 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:01.289002 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:01.289085 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:01.291927 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:49261: BlockingRecv error: recv got EOF from 127.19.53.194:49261 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 13:43:01 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:01 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:01.418404 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:01.413659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:39433)
0924 13:43:01.413818 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:01.413830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:01.413945 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:01.414100 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:01.414100 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:01.414278 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:01.414324 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:01.414326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:01.415020 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:01.415021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:01.415566 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:01.415567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:01.415958 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:01.415959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:01.416021 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:01.416499 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:01.416503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:01.418320 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
W20260924 13:43:01.418463 28846 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:01.413733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:39433 (local address 127.19.53.254:38999)
0924 13:43:01.413813 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:01.413815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:01.413903 (+ 88us) server_negotiation.cc:407] Connection header received
0924 13:43:01.414028 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:01.414029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:01.414042 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:01.414084 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:01.414422 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:01.414929 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:01.415656 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:01.415863 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:01.415955 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:01.418416 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:39433: BlockingRecv error: recv got EOF from 127.19.53.195:39433 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 13:43:02 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:02 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:02.150296 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:02.145862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:47133)
0924 13:43:02.146036 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:02.146047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:02.146110 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:02.146252 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:02.146253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:02.146420 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:02.146470 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:02.146472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:02.147104 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:02.147104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:02.147633 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:02.147634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:02.147967 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:02.147968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:02.148016 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:02.148456 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:02.148460 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:02.150237 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":109,"threads_started":1}
W20260924 13:43:02.150449 28850 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:02.145953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:47133 (local address 127.19.53.254:38999)
0924 13:43:02.146056 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:02.146058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:02.146094 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:43:02.146169 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:02.146170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:02.146180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:02.146238 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:02.146539 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:02.147040 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:02.147710 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:02.147903 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:02.147957 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:02.150328 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:47133: BlockingRecv error: recv got EOF from 127.19.53.193:47133 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":47,"threads_started":1}
Sep 24 13:43:02 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:02 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:02.309218 28851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:02.292415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:44675)
0924 13:43:02.292561 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:02.292571 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:02.292626 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:02.292749 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:02.292750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:02.292921 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:02.292970 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:02.292972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:02.293712 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:02.293713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:02.294253 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:02.294255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:02.294580 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:02.294581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:02.294620 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:02.307339 (+ 12719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:02.307344 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:02.309141 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
W20260924 13:43:02.309317 28850 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:02.292475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:44675 (local address 127.19.53.254:38999)
0924 13:43:02.292561 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:02.292562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:02.292629 (+ 67us) server_negotiation.cc:407] Connection header received
0924 13:43:02.292667 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:02.292667 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:02.292680 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:02.292731 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:02.293137 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:02.293622 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:02.294320 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:02.294516 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:02.294569 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:02.309247 (+ 14678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:44675: BlockingRecv error: recv got EOF from 127.19.53.194:44675 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 13:43:02 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:02 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:02.423305 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:02.418875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:42569)
0924 13:43:02.419040 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:02.419050 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:02.419104 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:02.419232 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:02.419232 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:02.419395 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:02.419443 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:02.419445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:02.420049 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:02.420050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:02.420564 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:02.420565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:02.420867 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:02.420868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:02.420906 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:02.421432 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:02.421437 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:02.423246 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
W20260924 13:43:02.423460 28850 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:02.418976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:42569 (local address 127.19.53.254:38999)
0924 13:43:02.419035 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:02.419038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:02.419115 (+ 77us) server_negotiation.cc:407] Connection header received
0924 13:43:02.419156 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:02.419156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:02.419169 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:02.419213 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:02.419502 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:02.419985 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:02.420629 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:02.420803 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:02.420851 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:02.423356 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:42569: BlockingRecv error: recv got EOF from 127.19.53.195:42569 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 13:43:03 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:03 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:03.155246 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:03.150779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:49899)
0924 13:43:03.150915 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:03.150927 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:03.150999 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:03.151133 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:03.151133 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:03.151293 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:03.151343 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:03.151344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:03.151965 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:03.151965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:03.152476 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:03.152477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:03.152782 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:03.152783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:03.152820 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:03.153273 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:03.153278 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:03.155164 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":54,"threads_started":1}
W20260924 13:43:03.155407 28854 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:03.150871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:49899 (local address 127.19.53.254:38999)
0924 13:43:03.151010 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:03.151012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:03.151020 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:43:03.151041 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:03.151042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:03.151051 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:03.151111 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:03.151406 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:03.151901 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:03.152542 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:03.152723 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:03.152770 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:03.155297 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:49899: BlockingRecv error: recv got EOF from 127.19.53.193:49899 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
Sep 24 13:43:03 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:03 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:03.314095 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:03.309734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:60153)
0924 13:43:03.309898 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:03.309923 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:03.309978 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:03.310115 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:03.310115 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:03.310272 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:03.310319 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:03.310321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:03.310967 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:03.310967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:03.311602 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:03.311605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:03.311907 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:03.311908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:03.311947 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:03.312340 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:03.312344 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:03.314014 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
W20260924 13:43:03.314209 28854 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:03.309806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:60153 (local address 127.19.53.254:38999)
0924 13:43:03.309887 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:03.309889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:03.309967 (+ 78us) server_negotiation.cc:407] Connection header received
0924 13:43:03.310035 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:03.310036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:03.310047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:03.310106 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:03.310378 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:03.310876 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:03.311663 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:03.311848 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:03.311895 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:03.314112 (+ 2217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:60153: BlockingRecv error: recv got EOF from 127.19.53.194:60153 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 13:43:03 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:03 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:03.428452 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:03.423781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:36169)
0924 13:43:03.423917 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:03.423930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:03.423992 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:03.424147 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:03.424148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:03.424328 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:03.424373 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:03.424375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:03.425062 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:03.425063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:03.425617 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:03.425619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:03.426041 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:03.426042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:03.426109 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:03.426627 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:03.426639 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:03.428392 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
W20260924 13:43:03.428550 28854 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:03.423924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:36169 (local address 127.19.53.254:38999)
0924 13:43:03.423971 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:03.423973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:03.423981 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:43:03.424058 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:03.424059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:03.424072 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:03.424140 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:03.424472 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:03.424976 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:03.425703 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:03.425951 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:03.426038 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:03.428482 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:36169: BlockingRecv error: recv got EOF from 127.19.53.195:36169 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 13:43:04 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:04 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:04.160372 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:04.155749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:48761)
0924 13:43:04.155924 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:04.155936 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:04.156002 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:04.156162 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:04.156163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:04.156327 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:04.156378 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:04.156380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:04.157038 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:04.157038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:04.157577 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:04.157578 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:04.157958 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:04.157959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:04.158004 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:04.158553 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:04.158558 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:04.160300 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":106,"threads_started":1}
W20260924 13:43:04.160513 28858 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:04.155849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:48761 (local address 127.19.53.254:38999)
0924 13:43:04.155953 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:04.155955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:04.155986 (+ 31us) server_negotiation.cc:407] Connection header received
0924 13:43:04.156073 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:04.156074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:04.156084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:04.156140 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:04.156445 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:04.156969 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:04.157647 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:04.157859 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:04.157952 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:04.160401 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:48761: BlockingRecv error: recv got EOF from 127.19.53.193:48761 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":49,"threads_started":1}
Sep 24 13:43:04 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:04 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:04.319036 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:04.314619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:46151)
0924 13:43:04.314765 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:04.314777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:04.314841 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:04.314966 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:04.314967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:04.315153 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:04.315211 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:04.315213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:04.315841 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:04.315842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:04.316339 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:04.316340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:04.316691 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:04.316691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:04.316743 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:04.317187 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:04.317191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:04.318968 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
W20260924 13:43:04.319150 28858 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:04.314665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:46151 (local address 127.19.53.254:38999)
0924 13:43:04.314760 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:04.314762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:04.314820 (+ 58us) server_negotiation.cc:407] Connection header received
0924 13:43:04.314895 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:04.314896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:04.314909 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:04.314951 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:04.315289 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:04.315756 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:04.316424 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:04.316616 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:04.316691 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:04.319076 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:46151: BlockingRecv error: recv got EOF from 127.19.53.194:46151 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 13:43:04 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:04 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:04.433580 28860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:04.428946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:56751)
0924 13:43:04.429080 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:04.429094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:04.429156 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:04.429280 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:04.429281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:04.429455 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:04.429503 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:04.429505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:04.430225 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:04.430226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:04.430765 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:04.430766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:04.431094 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:04.431095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:04.431132 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:04.431601 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:04.431605 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:04.433454 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
W20260924 13:43:04.433642 28858 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:04.429043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:56751 (local address 127.19.53.254:38999)
0924 13:43:04.429105 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:04.429107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:04.429150 (+ 43us) server_negotiation.cc:407] Connection header received
0924 13:43:04.429200 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:04.429200 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:04.429214 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:04.429265 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:04.429605 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:04.430150 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:04.430842 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:04.431027 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:04.431086 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:04.433572 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:56751: BlockingRecv error: recv got EOF from 127.19.53.195:56751 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 13:43:05 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:05 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:05.165500 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:05.160842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:57827)
0924 13:43:05.160996 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:05.161007 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:05.161062 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:05.161232 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:05.161232 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:05.161399 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:05.161445 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:05.161447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:05.162164 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:05.162165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:05.162688 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:05.162689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:05.163053 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:05.163053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:05.163092 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:05.163551 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:05.163555 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:05.165441 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
W20260924 13:43:05.165659 28862 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:05.160946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:57827 (local address 127.19.53.254:38999)
0924 13:43:05.161091 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:05.161093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:05.161101 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:43:05.161123 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:05.161124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:05.161136 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:05.161205 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:05.161515 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:05.162084 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:05.162768 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:05.162972 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:05.163045 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:05.165561 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:57827: BlockingRecv error: recv got EOF from 127.19.53.193:57827 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
Sep 24 13:43:05 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:05 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:05.324499 28863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:05.319510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:34829)
0924 13:43:05.319674 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:05.319685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:05.319751 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:05.319995 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:05.319996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:05.320182 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:05.320272 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:05.320274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:05.320960 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:05.320961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:05.321501 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:05.321503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:05.321827 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:05.321827 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:05.321867 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:05.322547 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:05.322553 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:05.324440 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
W20260924 13:43:05.324632 28862 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:05.319600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:34829 (local address 127.19.53.254:38999)
0924 13:43:05.319646 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:05.319648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:05.319738 (+ 90us) server_negotiation.cc:407] Connection header received
0924 13:43:05.319811 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:05.319811 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:05.319822 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:05.319867 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:05.320372 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:05.320881 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:05.321573 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:05.321746 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:05.321814 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:05.324548 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:34829: BlockingRecv error: recv got EOF from 127.19.53.194:34829 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 13:43:05 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:05 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:05.439065 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:05.434089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:41829)
0924 13:43:05.434251 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:05.434263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:05.434402 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:05.434574 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:05.434575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:05.434819 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:05.434870 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:05.434872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:05.435571 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:05.435572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:05.436443 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:05.436446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:05.436757 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:05.436758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:05.436794 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:05.437210 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:05.437214 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:05.439005 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
W20260924 13:43:05.439245 28862 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:05.434170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:41829 (local address 127.19.53.254:38999)
0924 13:43:05.434242 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:05.434244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:05.434373 (+ 129us) server_negotiation.cc:407] Connection header received
0924 13:43:05.434478 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:05.434479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:05.434493 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:05.434569 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:05.434950 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:05.435480 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:05.436511 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:05.436690 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:05.436742 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:05.439176 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:41829: BlockingRecv error: recv got EOF from 127.19.53.195:41829 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 13:43:06 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:06 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:06.170557 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:06.166006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:33233)
0924 13:43:06.166135 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:06.166146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:06.166201 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:06.166351 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:06.166351 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:06.166508 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:06.166555 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:06.166556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:06.167216 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:06.167218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:06.167786 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:06.167787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:06.168144 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:06.168145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:06.168192 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:06.168758 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:06.168763 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:06.170479 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
W20260924 13:43:06.170730 28866 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:06.166057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:33233 (local address 127.19.53.254:38999)
0924 13:43:06.166193 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:06.166195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:06.166205 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:43:06.166248 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:06.166249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:06.166264 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:06.166333 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:06.166623 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:06.167125 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:06.167866 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:06.168072 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:06.168133 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:06.170634 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:33233: BlockingRecv error: recv got EOF from 127.19.53.193:33233 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
Sep 24 13:43:06 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:06 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:06.329386 28867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:06.324922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:47595)
0924 13:43:06.325051 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:06.325063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:06.325129 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:06.325308 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:06.325308 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:06.325480 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:06.325527 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:06.325529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:06.326234 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:06.326235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:06.326801 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:06.326803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:06.327126 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:06.327127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:06.327178 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:06.327600 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:06.327604 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:06.329303 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
W20260924 13:43:06.329437 28866 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:06.325009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:47595 (local address 127.19.53.254:38999)
0924 13:43:06.325057 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:06.325059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:06.325115 (+ 56us) server_negotiation.cc:407] Connection header received
0924 13:43:06.325173 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:06.325174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:06.325184 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:06.325301 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:06.325595 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:06.326170 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:06.326878 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:06.327065 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:06.327115 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:06.329392 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:47595: BlockingRecv error: recv got EOF from 127.19.53.194:47595 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 13:43:06 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:06 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:06.444383 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:06.439513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:55211)
0924 13:43:06.439700 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:06.439712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:06.439801 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:06.439993 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:06.439994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:06.440178 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:06.440222 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:06.440224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:06.440921 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:06.440922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:06.441453 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:06.441455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:06.441847 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:06.441848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:06.441955 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:06.442522 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:06.442527 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:06.444280 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":97,"threads_started":1}
W20260924 13:43:06.444435 28866 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:06.439599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:55211 (local address 127.19.53.254:38999)
0924 13:43:06.439684 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:06.439687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:06.439788 (+ 101us) server_negotiation.cc:407] Connection header received
0924 13:43:06.439898 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:06.439899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:06.439914 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:06.439987 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:06.440353 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:06.440837 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:06.441540 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:06.441757 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:06.441826 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:06.444370 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:55211: BlockingRecv error: recv got EOF from 127.19.53.195:55211 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 13:43:07 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:07 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:07.177038 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:07.170981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:40115)
0924 13:43:07.171093 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:07.171104 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:07.171164 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:07.171302 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:07.171303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:07.171464 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:07.171539 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:07.171541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:07.172670 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:07.172671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:07.173759 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:07.173762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:07.174347 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:07.174348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:07.174407 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:07.174956 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:07.174961 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:07.176979 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":51,"threads_started":1}
W20260924 13:43:07.177206 28870 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:07.171026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:40115 (local address 127.19.53.254:38999)
0924 13:43:07.171159 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:07.171161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:07.171172 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:43:07.171225 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:07.171226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:07.171236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:07.171305 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:07.171612 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:07.172524 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:07.173854 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:07.174254 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:07.174331 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:07.177097 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:40115: BlockingRecv error: recv got EOF from 127.19.53.193:40115 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
Sep 24 13:43:07 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:07 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:07.341792 28871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:07.329877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:54109)
0924 13:43:07.330039 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:07.330049 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:07.330129 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:07.330297 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:07.330299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:07.330475 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:07.330522 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:07.330524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:07.331198 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:07.331199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:07.331711 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:07.331713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:07.332047 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:07.332048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:07.332109 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:07.339670 (+ 7561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:07.339675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:07.341686 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
W20260924 13:43:07.341840 28870 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:07.329938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:54109 (local address 127.19.53.254:38999)
0924 13:43:07.329998 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:07.330000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:07.330110 (+ 110us) server_negotiation.cc:407] Connection header received
0924 13:43:07.330207 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:07.330208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:07.330222 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:07.330292 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:07.330602 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:07.331118 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:07.331792 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:07.331979 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:07.332049 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:07.341774 (+ 9725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:54109: BlockingRecv error: recv got EOF from 127.19.53.194:54109 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 13:43:07 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:07 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.195@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:07.449628 28872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:07.444922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.195:33387)
0924 13:43:07.445095 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:07.445113 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:07.445193 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:07.445359 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:07.445360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:07.445544 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:07.445589 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:07.445591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:07.446357 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:07.446362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:07.446885 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:07.446887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:07.447231 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:07.447232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:07.447289 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:07.447774 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:07.447779 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:07.449559 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
W20260924 13:43:07.449693 28870 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:07.444999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:33387 (local address 127.19.53.254:38999)
0924 13:43:07.445076 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:07.445078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:07.445174 (+ 96us) server_negotiation.cc:407] Connection header received
0924 13:43:07.445269 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:07.445270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:07.445284 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:07.445352 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:07.445673 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:07.446248 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:07.446976 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:07.447162 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:07.447234 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:07.449647 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.195:33387: BlockingRecv error: recv got EOF from 127.19.53.195:33387 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 13:43:08 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:08 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.193@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:08.182595 28873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:08.177947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.193:58659)
0924 13:43:08.178084 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:08.178103 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:08.178173 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:08.178346 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:08.178346 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:08.178504 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:08.178554 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:08.178556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.179180 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.179180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:08.179705 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:08.179706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.180039 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.180040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:08.180080 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:08.180601 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:08.180606 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:08.182514 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
W20260924 13:43:08.182741 28874 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:08.178026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:58659 (local address 127.19.53.254:38999)
0924 13:43:08.178164 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:08.178166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:08.178178 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:43:08.178250 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:08.178251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:08.178262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:08.178320 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:08.178624 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.179115 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.179778 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.179975 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.180028 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:08.182616 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.193:58659: BlockingRecv error: recv got EOF from 127.19.53.193:58659 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
Sep 24 13:43:08 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:43:08 dist-test-slave-tfm9 krb5kdc[27805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.53.194@KRBTEST.COM for oryx/127.19.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:43:08.347050 28875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:08.342337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38999 (local address 127.19.53.194:43973)
0924 13:43:08.342475 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:08.342488 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:08.342556 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:08.342699 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:08.342700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:08.342971 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:08.343020 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:08.343022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.343929 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.343930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:08.344470 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:08.344472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.344788 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.344789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:08.344831 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:08.345252 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:08.345257 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:08.346992 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.53.254:38999: Server oryx/127.19.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
W20260924 13:43:08.347154 28874 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:08.342410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:43973 (local address 127.19.53.254:38999)
0924 13:43:08.342505 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:08.342507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:08.342550 (+ 43us) server_negotiation.cc:407] Connection header received
0924 13:43:08.342613 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:08.342614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:08.342625 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:08.342674 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:08.343089 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.343814 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.344539 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.344721 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.344771 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:08.347083 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.53.194:43973: BlockingRecv error: recv got EOF from 127.19.53.194:43973 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 13:43:08.364738 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 28393
I20260924 13:43:08.369504 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 28547
I20260924 13:43:08.373838 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 28682
I20260924 13:43:08.377827 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 28322
2026-09-24T13:43:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17411 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T13:43:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:43:08Z Disabled control of system clock
I20260924 13:43:08.405427 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:45007
--webserver_interface=127.19.53.254
--webserver_port=0
--builtin_ntp_servers=127.19.53.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:45007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 13:43:08.488557 28879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:08.488775 28879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:08.488798 28879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:08.489940 28879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:43:08.489982 28879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:08.489996 28879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:08.490011 28879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:43:08.490024 28879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:43:08.491298 28879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:45007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:45007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:08.491500 28879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:08.491772 28879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:43:08.496965 28879 server_base.cc:1034] running on GCE node
W20260924 13:43:08.496937 28884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:08.497000 28887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:08.496943 28885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:08.497392 28879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:08.497714 28879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:08.498859 28879 hybrid_clock.cc:648] HybridClock initialized: now 1790257388498841 us; error 40 us; skew 500 ppm
I20260924 13:43:08.500041 28879 webserver.cc:460] Webserver started at http://127.19.53.254:38387/ using document root <none> and password file <none>
I20260924 13:43:08.500295 28879 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:08.500365 28879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:08.500542 28879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:43:08.501598 28879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "412d537b70b5404ca9ffee9b1da42695"
format_stamp: "Formatted at 2026-09-24 13:43:08 on dist-test-slave-tfm9"
server_key: "e6ddbffb9dd155d0c7aecad6cd5e6a63"
server_key_iv: "52013274e57fad2ee173c604090757f2"
server_key_version: "encryptionkey@0"
I20260924 13:43:08.502022 28879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "412d537b70b5404ca9ffee9b1da42695"
format_stamp: "Formatted at 2026-09-24 13:43:08 on dist-test-slave-tfm9"
server_key: "e6ddbffb9dd155d0c7aecad6cd5e6a63"
server_key_iv: "52013274e57fad2ee173c604090757f2"
server_key_version: "encryptionkey@0"
I20260924 13:43:08.503307 28879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:43:08.504053 28893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:08.504225 28879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:43:08.504323 28879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "412d537b70b5404ca9ffee9b1da42695"
format_stamp: "Formatted at 2026-09-24 13:43:08 on dist-test-slave-tfm9"
server_key: "e6ddbffb9dd155d0c7aecad6cd5e6a63"
server_key_iv: "52013274e57fad2ee173c604090757f2"
server_key_version: "encryptionkey@0"
I20260924 13:43:08.504375 28879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:08.526320 28879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:08.526609 28879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:08.526712 28879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:08.530441 28879 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:45007
I20260924 13:43:08.530501 28946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:45007 every 8 connection(s)
I20260924 13:43:08.530839 28879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:43:08.531539 28947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:43:08.534147 28947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695: Bootstrap starting.
I20260924 13:43:08.534731 28947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695: Neither blocks nor log segments found. Creating new log.
I20260924 13:43:08.535035 28947 log.cc:824] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695: Log is configured to *not* fsync() on all Append() calls
I20260924 13:43:08.535723 28947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695: No bootstrap required, opened a new log
I20260924 13:43:08.537267 28947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412d537b70b5404ca9ffee9b1da42695" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 45007 } }
I20260924 13:43:08.537429 28947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:43:08.537479 28947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412d537b70b5404ca9ffee9b1da42695, State: Initialized, Role: FOLLOWER
I20260924 13:43:08.537647 28947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412d537b70b5404ca9ffee9b1da42695" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 45007 } }
I20260924 13:43:08.537730 28947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:43:08.537774 28947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:43:08.537830 28947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:43:08.538396 28947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412d537b70b5404ca9ffee9b1da42695" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 45007 } }
I20260924 13:43:08.538516 28947 leader_election.cc:302] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 412d537b70b5404ca9ffee9b1da42695; no voters:
I20260924 13:43:08.538698 28947 leader_election.cc:288] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:43:08.538749 28950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:43:08.538898 28950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [term 1 LEADER]: Becoming Leader. State: Replica: 412d537b70b5404ca9ffee9b1da42695, State: Running, Role: LEADER
I20260924 13:43:08.538976 28947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:43:08.539057 28950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412d537b70b5404ca9ffee9b1da42695" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 45007 } }
I20260924 13:43:08.539525 28952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412d537b70b5404ca9ffee9b1da42695. Latest consensus state: current_term: 1 leader_uuid: "412d537b70b5404ca9ffee9b1da42695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412d537b70b5404ca9ffee9b1da42695" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 45007 } } }
I20260924 13:43:08.539614 28952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [sys.catalog]: This master's current role is: LEADER
I20260924 13:43:08.539654 28951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "412d537b70b5404ca9ffee9b1da42695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412d537b70b5404ca9ffee9b1da42695" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 45007 } } }
I20260924 13:43:08.539717 28951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695 [sys.catalog]: This master's current role is: LEADER
I20260924 13:43:08.539870 28959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:43:08.539904 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 28879
I20260924 13:43:08.540021 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:43:08.540210 19671 external_mini_cluster.cc:1428] Setting key ccf795d1b7fb7ffaed84e0fce7744049
I20260924 13:43:08.540336 28959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:43:08.542054 28959 catalog_manager.cc:1347] Generated new cluster ID: e6dcc00c4bed45258d1b8a40e6a57d6c
I20260924 13:43:08.542111 28959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:43:08.550323 28959 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:43:08.550822 28959 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:43:08.555065 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:08.540659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.19.53.254:45007)
0924 13:43:08.541288 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:08.541297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:08.541327 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:43:08.541436 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:08.541443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:08.541510 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:08.549865 (+ 8355us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:08.550024 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.551538 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.552638 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.553273 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.553403 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:08.554186 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:08.554190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:08.554190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:43:08.554409 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:08.554452 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:43:08.554467 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:08.554536 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:08.554618 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 13:43:08.554833 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":75,"threads_started":1}
I20260924 13:43:08.559422 28959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 412d537b70b5404ca9ffee9b1da42695: Generated new TSK 0
I20260924 13:43:08.559623 28959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:43:08.561232 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:45007
--builtin_ntp_servers=127.19.53.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 13:43:08.643989 28971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:08.644181 28971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:08.644202 28971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:08.645377 28971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:08.645417 28971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:08.645442 28971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:43:08.646826 28971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:08.647074 28971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:08.647380 28971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:08.652282 28979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:08.652311 28977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:08.652374 28971 server_base.cc:1034] running on GCE node
W20260924 13:43:08.652429 28976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:08.652671 28971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:08.652935 28971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:08.654102 28971 hybrid_clock.cc:648] HybridClock initialized: now 1790257388654088 us; error 29 us; skew 500 ppm
I20260924 13:43:08.655421 28971 webserver.cc:460] Webserver started at http://127.19.53.193:41101/ using document root <none> and password file <none>
I20260924 13:43:08.655671 28971 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:08.655740 28971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:08.655896 28971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:43:08.656924 28971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "875132b9e556416aa34bd22b307552d0"
format_stamp: "Formatted at 2026-09-24 13:43:08 on dist-test-slave-tfm9"
server_key: "c88d27888099a59374687d43d8bf3bd9"
server_key_iv: "f0a4dba5c8471d0e226985a31f890ff9"
server_key_version: "encryptionkey@0"
I20260924 13:43:08.657279 28971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "875132b9e556416aa34bd22b307552d0"
format_stamp: "Formatted at 2026-09-24 13:43:08 on dist-test-slave-tfm9"
server_key: "c88d27888099a59374687d43d8bf3bd9"
server_key_iv: "f0a4dba5c8471d0e226985a31f890ff9"
server_key_version: "encryptionkey@0"
I20260924 13:43:08.658692 28971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:43:08.659449 28985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:08.659694 28971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:43:08.659801 28971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "875132b9e556416aa34bd22b307552d0"
format_stamp: "Formatted at 2026-09-24 13:43:08 on dist-test-slave-tfm9"
server_key: "c88d27888099a59374687d43d8bf3bd9"
server_key_iv: "f0a4dba5c8471d0e226985a31f890ff9"
server_key_version: "encryptionkey@0"
I20260924 13:43:08.659873 28971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:08.678997 28971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:08.679258 28971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:08.679366 28971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:08.679538 28971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:43:08.679880 28971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:43:08.679914 28971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:08.679936 28971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:43:08.679983 28971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:08.685434 28971 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:37241
I20260924 13:43:08.685511 29099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:37241 every 8 connection(s)
I20260924 13:43:08.685851 28971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:43:08.686424 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 28971
I20260924 13:43:08.686492 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:43:08.686723 19671 external_mini_cluster.cc:1428] Setting key e2a70da2aab38fb95e425769f29511f3
I20260924 13:43:08.687759 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:45007
--builtin_ntp_servers=127.19.53.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 13:43:08.691463 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:08.686635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:56997 (local address 127.19.53.254:45007)
0924 13:43:08.686688 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:08.686691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:08.686984 (+ 293us) server_negotiation.cc:407] Connection header received
0924 13:43:08.687430 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:08.687431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:08.687445 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:08.687488 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:08.688394 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.688907 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.689906 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.690138 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.690229 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:08.691095 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:08.691097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:08.691097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:43:08.691102 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:08.691119 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:43:08.691128 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:08.691190 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:08.691304 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 13:43:08.691343 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:43:08.691591 29102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:08.686639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:45007 (local address 127.19.53.193:56997)
0924 13:43:08.686844 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:08.686885 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:08.687359 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:08.687569 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:08.687573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:08.687584 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:08.688302 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:08.688310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.689021 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.689022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:08.689809 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:08.689812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.690241 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.690242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:08.690385 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:08.690957 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:43:08.690957 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:08.690986 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:43:08.691008 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:43:08.691190 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:43:08.691197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:43:08.691200 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:43:08.691274 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 13:43:08.691398 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260924 13:43:08.692147 29100 heartbeater.cc:347] Connected to a master server at 127.19.53.254:45007
I20260924 13:43:08.692255 29100 heartbeater.cc:464] Registering TS with master...
I20260924 13:43:08.692472 29100 heartbeater.cc:511] Master 127.19.53.254:45007 requested a full tablet report, sending...
I20260924 13:43:08.693114 28911 ts_manager.cc:194] Registered new tserver with Master: 875132b9e556416aa34bd22b307552d0 (127.19.53.193:37241)
I20260924 13:43:08.694134 28911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.53.193:56997
W20260924 13:43:08.770068 29104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:08.770270 29104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:08.770291 29104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:08.771478 29104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:08.771521 29104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:08.771546 29104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:43:08.772943 29104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:08.773177 29104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:08.773478 29104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:08.778378 29110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:08.778378 29112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:08.778345 29109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:08.778951 29104 server_base.cc:1034] running on GCE node
I20260924 13:43:08.779201 29104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:08.779462 29104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:08.780630 29104 hybrid_clock.cc:648] HybridClock initialized: now 1790257388780625 us; error 36 us; skew 500 ppm
I20260924 13:43:08.782117 29104 webserver.cc:460] Webserver started at http://127.19.53.194:32999/ using document root <none> and password file <none>
I20260924 13:43:08.782389 29104 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:08.782455 29104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:08.782596 29104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:43:08.783653 29104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "361b7da05fe9463eac7376e9c3959d20"
format_stamp: "Formatted at 2026-09-24 13:43:08 on dist-test-slave-tfm9"
server_key: "a6d82cee31ed9e26d8c8a467ddf9f0af"
server_key_iv: "3c7911b4e7d24e9ff31bff562d6e4fe4"
server_key_version: "encryptionkey@0"
I20260924 13:43:08.784019 29104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "361b7da05fe9463eac7376e9c3959d20"
format_stamp: "Formatted at 2026-09-24 13:43:08 on dist-test-slave-tfm9"
server_key: "a6d82cee31ed9e26d8c8a467ddf9f0af"
server_key_iv: "3c7911b4e7d24e9ff31bff562d6e4fe4"
server_key_version: "encryptionkey@0"
I20260924 13:43:08.785276 29104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:43:08.786337 29118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:08.786522 29104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:08.786631 29104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "361b7da05fe9463eac7376e9c3959d20"
format_stamp: "Formatted at 2026-09-24 13:43:08 on dist-test-slave-tfm9"
server_key: "a6d82cee31ed9e26d8c8a467ddf9f0af"
server_key_iv: "3c7911b4e7d24e9ff31bff562d6e4fe4"
server_key_version: "encryptionkey@0"
I20260924 13:43:08.786723 29104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:08.802114 29104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:08.802455 29104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:08.802644 29104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:08.802883 29104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:43:08.803248 29104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:43:08.803308 29104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:08.803354 29104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:43:08.803387 29104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:08.810062 29104 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:41919
I20260924 13:43:08.810103 29232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:41919 every 8 connection(s)
I20260924 13:43:08.810496 29104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:43:08.812920 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 29104
I20260924 13:43:08.813040 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:43:08.813257 19671 external_mini_cluster.cc:1428] Setting key 8cf206c41bc7b40cf2e28e4df7d3da85
I20260924 13:43:08.814396 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:45007
--builtin_ntp_servers=127.19.53.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 13:43:08.816260 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:08.811253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:44395 (local address 127.19.53.254:45007)
0924 13:43:08.811356 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:08.811359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:08.811565 (+ 206us) server_negotiation.cc:407] Connection header received
0924 13:43:08.812122 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:08.812123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:08.812136 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:08.812195 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:08.812886 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.813495 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.814696 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.814902 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.814990 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:08.815928 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:08.815930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:08.815930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:43:08.815935 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:08.815953 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:43:08.815976 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:08.816046 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:08.816119 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 13:43:08.816157 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:43:08.816460 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:08.811233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:45007 (local address 127.19.53.194:44395)
0924 13:43:08.811443 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:08.811476 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:08.812003 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:08.812266 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:08.812270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:08.812280 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:08.812780 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:08.812784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.813584 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.813585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:08.814609 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:08.814611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.815001 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.815002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:08.815144 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:08.815765 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:43:08.815765 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:08.815794 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:43:08.815818 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:43:08.816054 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:43:08.816058 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:43:08.816061 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:43:08.816118 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 13:43:08.816201 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260924 13:43:08.816931 29233 heartbeater.cc:347] Connected to a master server at 127.19.53.254:45007
I20260924 13:43:08.817031 29233 heartbeater.cc:464] Registering TS with master...
I20260924 13:43:08.817220 29233 heartbeater.cc:511] Master 127.19.53.254:45007 requested a full tablet report, sending...
I20260924 13:43:08.817600 28911 ts_manager.cc:194] Registered new tserver with Master: 361b7da05fe9463eac7376e9c3959d20 (127.19.53.194:41919)
I20260924 13:43:08.818152 28911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.53.194:44395
W20260924 13:43:08.899050 29236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:08.899243 29236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:08.899263 29236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:08.900338 29236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:08.900370 29236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:08.900394 29236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:43:08.901594 29236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:08.901763 29236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:08.902081 29236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:08.907002 29244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:08.907058 29241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:08.907011 29242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:08.907284 29236 server_base.cc:1034] running on GCE node
I20260924 13:43:08.907518 29236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:08.907757 29236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:08.908900 29236 hybrid_clock.cc:648] HybridClock initialized: now 1790257388908889 us; error 27 us; skew 500 ppm
I20260924 13:43:08.910305 29236 webserver.cc:460] Webserver started at http://127.19.53.195:35565/ using document root <none> and password file <none>
I20260924 13:43:08.910569 29236 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:08.910627 29236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:08.910818 29236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:43:08.911978 29236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "56b15b2077c342f19f23e6e7e5ab6280"
format_stamp: "Formatted at 2026-09-24 13:43:08 on dist-test-slave-tfm9"
server_key: "2b6c0866bd78d30b1d538f0bb9147818"
server_key_iv: "c7555d6e103d9a3be2760abf5da6b275"
server_key_version: "encryptionkey@0"
I20260924 13:43:08.912365 29236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "56b15b2077c342f19f23e6e7e5ab6280"
format_stamp: "Formatted at 2026-09-24 13:43:08 on dist-test-slave-tfm9"
server_key: "2b6c0866bd78d30b1d538f0bb9147818"
server_key_iv: "c7555d6e103d9a3be2760abf5da6b275"
server_key_version: "encryptionkey@0"
I20260924 13:43:08.913775 29236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:43:08.914628 29250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:08.914798 29236 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:43:08.914882 29236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "56b15b2077c342f19f23e6e7e5ab6280"
format_stamp: "Formatted at 2026-09-24 13:43:08 on dist-test-slave-tfm9"
server_key: "2b6c0866bd78d30b1d538f0bb9147818"
server_key_iv: "c7555d6e103d9a3be2760abf5da6b275"
server_key_version: "encryptionkey@0"
I20260924 13:43:08.914937 29236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:08.923803 29236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:08.924018 29236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:08.924108 29236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:08.924260 29236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:43:08.924508 29236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:43:08.924536 29236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:08.924556 29236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:43:08.924569 29236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:08.930617 29236 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:44453
I20260924 13:43:08.930697 29364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:44453 every 8 connection(s)
I20260924 13:43:08.930996 29236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:43:08.936087 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:08.931794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:57727 (local address 127.19.53.254:45007)
0924 13:43:08.931865 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:08.931867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:08.932088 (+ 221us) server_negotiation.cc:407] Connection header received
0924 13:43:08.932573 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:08.932574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:08.932587 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:08.932644 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:08.933300 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.933774 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.934808 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.934992 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.935044 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:08.935851 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:08.935853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:08.935853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:43:08.935857 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:08.935870 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:43:08.935874 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:08.935916 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:08.936006 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 13:43:08.936030 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 13:43:08.936359 29367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:08.931778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:45007 (local address 127.19.53.195:57727)
0924 13:43:08.931960 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:08.931999 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:08.932491 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:08.932695 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:08.932698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:08.932706 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:08.933227 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:08.933232 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.933857 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.933858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:08.934728 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:08.934730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:08.935070 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:08.935071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:08.935182 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:08.935704 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:43:08.935705 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:08.935735 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:43:08.935757 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:43:08.935932 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:43:08.935936 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:43:08.935938 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:43:08.935995 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 13:43:08.936072 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260924 13:43:08.936872 29365 heartbeater.cc:347] Connected to a master server at 127.19.53.254:45007
I20260924 13:43:08.936971 29365 heartbeater.cc:464] Registering TS with master...
I20260924 13:43:08.937165 29365 heartbeater.cc:511] Master 127.19.53.254:45007 requested a full tablet report, sending...
I20260924 13:43:08.937567 28911 ts_manager.cc:194] Registered new tserver with Master: 56b15b2077c342f19f23e6e7e5ab6280 (127.19.53.195:44453)
I20260924 13:43:08.937990 28911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.53.195:57727
I20260924 13:43:08.939648 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 29236
I20260924 13:43:08.939713 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:43:08.939934 19671 external_mini_cluster.cc:1428] Setting key 0146224c9752f9213779a521933e5232
I20260924 13:43:08.940790 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 13:43:08.942485 28963 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:08.942062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.19.53.254:45007)
0924 13:43:08.942092 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:08.942094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:08.942212 (+ 118us) server_negotiation.cc:407] Connection header received
0924 13:43:08.942257 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:08.942257 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:08.942265 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:08.942307 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:08.942414 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36848: BlockingRecv error: recv got EOF from 127.0.0.1:36848 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 13:43:08.942587 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 28971
I20260924 13:43:08.947582 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 29104
I20260924 13:43:08.952740 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 29236
I20260924 13:43:08.956867 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 28879
2026-09-24T13:43:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (577 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T13:43:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:43:08Z Disabled control of system clock
I20260924 13:43:08.982908 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33313
--webserver_interface=127.19.53.254
--webserver_port=0
--builtin_ntp_servers=127.19.53.212:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:33313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 13:43:09.062096 29376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:09.062292 29376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:09.062314 29376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:09.063460 29376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:43:09.063503 29376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:09.063514 29376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:09.063527 29376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:43:09.063580 29376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:43:09.064899 29376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:33313
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:09.065158 29376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:09.065472 29376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:09.070292 29381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:09.070286 29382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:09.070308 29384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:09.070497 29376 server_base.cc:1034] running on GCE node
I20260924 13:43:09.070752 29376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:09.071101 29376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:09.072259 29376 hybrid_clock.cc:648] HybridClock initialized: now 1790257389072238 us; error 36 us; skew 500 ppm
I20260924 13:43:09.073542 29376 webserver.cc:460] Webserver started at http://127.19.53.254:39041/ using document root <none> and password file <none>
I20260924 13:43:09.073798 29376 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:09.073874 29376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:09.074054 29376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:43:09.075157 29376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "9990f12ad7614e62b5d1d449df81a2f7"
format_stamp: "Formatted at 2026-09-24 13:43:09 on dist-test-slave-tfm9"
server_key: "cda22ee2c3b519ee9baf0e95975fea39"
server_key_iv: "fa91258a67afdad35eb8e67576a5b232"
server_key_version: "encryptionkey@0"
I20260924 13:43:09.075520 29376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "9990f12ad7614e62b5d1d449df81a2f7"
format_stamp: "Formatted at 2026-09-24 13:43:09 on dist-test-slave-tfm9"
server_key: "cda22ee2c3b519ee9baf0e95975fea39"
server_key_iv: "fa91258a67afdad35eb8e67576a5b232"
server_key_version: "encryptionkey@0"
I20260924 13:43:09.076762 29376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:43:09.077491 29390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:09.077702 29376 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:43:09.077797 29376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "9990f12ad7614e62b5d1d449df81a2f7"
format_stamp: "Formatted at 2026-09-24 13:43:09 on dist-test-slave-tfm9"
server_key: "cda22ee2c3b519ee9baf0e95975fea39"
server_key_iv: "fa91258a67afdad35eb8e67576a5b232"
server_key_version: "encryptionkey@0"
I20260924 13:43:09.077879 29376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:09.081082 29376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:09.081374 29376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:09.081532 29376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:09.085363 29376 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:33313
I20260924 13:43:09.085410 29443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:33313 every 8 connection(s)
I20260924 13:43:09.085733 29376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:43:09.086441 29444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:43:09.086747 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 29376
I20260924 13:43:09.086835 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:43:09.087035 19671 external_mini_cluster.cc:1428] Setting key e78804c8e99f33c4b18524bfbd75c013
I20260924 13:43:09.089838 29447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:09.087632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.19.53.254:33313)
0924 13:43:09.087829 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:09.087837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:09.087872 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:43:09.088520 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:09.088530 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:09.088585 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:09.088746 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:09.089389 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:09.089392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:09.089393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:43:09.089397 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:09.089456 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:43:09.089470 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:09.089545 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:09.089596 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 13:43:09.089650 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260924 13:43:09.090324 29444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7: Bootstrap starting.
I20260924 13:43:09.090899 29444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7: Neither blocks nor log segments found. Creating new log.
I20260924 13:43:09.091184 29444 log.cc:824] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7: Log is configured to *not* fsync() on all Append() calls
I20260924 13:43:09.091827 29444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7: No bootstrap required, opened a new log
I20260924 13:43:09.093160 29444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9990f12ad7614e62b5d1d449df81a2f7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33313 } }
I20260924 13:43:09.093334 29444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:43:09.093405 29444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9990f12ad7614e62b5d1d449df81a2f7, State: Initialized, Role: FOLLOWER
I20260924 13:43:09.093564 29444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9990f12ad7614e62b5d1d449df81a2f7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33313 } }
I20260924 13:43:09.093657 29444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:43:09.093698 29444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:43:09.093743 29444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:43:09.094355 29444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9990f12ad7614e62b5d1d449df81a2f7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33313 } }
I20260924 13:43:09.094472 29444 leader_election.cc:302] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9990f12ad7614e62b5d1d449df81a2f7; no voters:
I20260924 13:43:09.094656 29444 leader_election.cc:288] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:43:09.094676 29449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:43:09.094817 29449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [term 1 LEADER]: Becoming Leader. State: Replica: 9990f12ad7614e62b5d1d449df81a2f7, State: Running, Role: LEADER
I20260924 13:43:09.094931 29444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:43:09.094949 29449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9990f12ad7614e62b5d1d449df81a2f7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33313 } }
I20260924 13:43:09.095324 29449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9990f12ad7614e62b5d1d449df81a2f7. Latest consensus state: current_term: 1 leader_uuid: "9990f12ad7614e62b5d1d449df81a2f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9990f12ad7614e62b5d1d449df81a2f7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33313 } } }
I20260924 13:43:09.095402 29449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [sys.catalog]: This master's current role is: LEADER
I20260924 13:43:09.095353 29451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9990f12ad7614e62b5d1d449df81a2f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9990f12ad7614e62b5d1d449df81a2f7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33313 } } }
I20260924 13:43:09.095471 29451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7 [sys.catalog]: This master's current role is: LEADER
I20260924 13:43:09.095621 29456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:43:09.096099 29456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:43:09.097630 29456 catalog_manager.cc:1347] Generated new cluster ID: c1ce53adcdf2498fbb4830505061d58c
I20260924 13:43:09.097679 29456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:43:09.107514 29456 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:43:09.107658 29456 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:43:09.120158 29456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9990f12ad7614e62b5d1d449df81a2f7: Generated new TSK 0
I20260924 13:43:09.120358 29456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:43:09.126935 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33313
--builtin_ntp_servers=127.19.53.212:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 13:43:09.207453 29468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:09.207643 29468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:09.207664 29468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:09.208809 29468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:09.208856 29468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:09.208881 29468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:43:09.210285 29468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:09.210505 29468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:09.210798 29468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:09.216351 29474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:09.216352 29476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:09.216334 29473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:09.216583 29468 server_base.cc:1034] running on GCE node
I20260924 13:43:09.216846 29468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:09.217151 29468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:09.218298 29468 hybrid_clock.cc:648] HybridClock initialized: now 1790257389218288 us; error 27 us; skew 500 ppm
I20260924 13:43:09.219619 29468 webserver.cc:460] Webserver started at http://127.19.53.193:41053/ using document root <none> and password file <none>
I20260924 13:43:09.219887 29468 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:09.219931 29468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:09.220062 29468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:43:09.221156 29468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "83be587ed16d486b8bc2b8ccc2218be0"
format_stamp: "Formatted at 2026-09-24 13:43:09 on dist-test-slave-tfm9"
server_key: "49d595ec16b1c5a886e58bf51b74be58"
server_key_iv: "450897d6b51a71191dcdea76dea3c674"
server_key_version: "encryptionkey@0"
I20260924 13:43:09.221509 29468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "83be587ed16d486b8bc2b8ccc2218be0"
format_stamp: "Formatted at 2026-09-24 13:43:09 on dist-test-slave-tfm9"
server_key: "49d595ec16b1c5a886e58bf51b74be58"
server_key_iv: "450897d6b51a71191dcdea76dea3c674"
server_key_version: "encryptionkey@0"
I20260924 13:43:09.222911 29468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:43:09.223587 29482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:09.223778 29468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:43:09.223867 29468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "83be587ed16d486b8bc2b8ccc2218be0"
format_stamp: "Formatted at 2026-09-24 13:43:09 on dist-test-slave-tfm9"
server_key: "49d595ec16b1c5a886e58bf51b74be58"
server_key_iv: "450897d6b51a71191dcdea76dea3c674"
server_key_version: "encryptionkey@0"
I20260924 13:43:09.223919 29468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:09.227088 29468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:09.227280 29468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:09.227371 29468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:09.227516 29468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:43:09.227731 29468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:43:09.227758 29468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:09.227777 29468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:43:09.227788 29468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:09.233191 29468 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:39539
I20260924 13:43:09.233259 29596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:39539 every 8 connection(s)
I20260924 13:43:09.233551 29468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:43:09.236470 29447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:09.234236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:54823 (local address 127.19.53.254:33313)
0924 13:43:09.234302 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:09.234303 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:09.234534 (+ 231us) server_negotiation.cc:407] Connection header received
0924 13:43:09.235001 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:09.235002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:09.235009 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:09.235067 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:09.236227 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:09.236228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:09.236228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:43:09.236233 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:09.236247 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:43:09.236252 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:09.236299 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:09.236389 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 13:43:09.236405 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 13:43:09.236691 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:09.234235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33313 (local address 127.19.53.193:54823)
0924 13:43:09.234420 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:09.234451 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:09.234899 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:09.235138 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:09.235142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:09.235150 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:09.236076 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:43:09.236076 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:09.236101 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:43:09.236127 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:43:09.236296 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:43:09.236300 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:43:09.236303 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:43:09.236398 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 13:43:09.236455 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260924 13:43:09.237221 29597 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33313
I20260924 13:43:09.237286 29597 heartbeater.cc:464] Registering TS with master...
I20260924 13:43:09.237455 29597 heartbeater.cc:511] Master 127.19.53.254:33313 requested a full tablet report, sending...
I20260924 13:43:09.237996 29408 ts_manager.cc:194] Registered new tserver with Master: 83be587ed16d486b8bc2b8ccc2218be0 (127.19.53.193:39539)
I20260924 13:43:09.241467 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 29468
I20260924 13:43:09.241564 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:43:09.241747 19671 external_mini_cluster.cc:1428] Setting key 63ffbfc63c9bef82accfa1df315e9472
I20260924 13:43:09.242568 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33313
--builtin_ntp_servers=127.19.53.212:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 13:43:09.320812 29600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:09.321053 29600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:09.321076 29600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:09.322221 29600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:09.322256 29600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:09.322281 29600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:43:09.323567 29600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:09.323734 29600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:09.323951 29600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:09.328683 29608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:09.328724 29606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:09.328691 29605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:09.329083 29600 server_base.cc:1034] running on GCE node
I20260924 13:43:09.329253 29600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:09.329432 29600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:09.330590 29600 hybrid_clock.cc:648] HybridClock initialized: now 1790257389330580 us; error 28 us; skew 500 ppm
I20260924 13:43:09.331733 29600 webserver.cc:460] Webserver started at http://127.19.53.194:44281/ using document root <none> and password file <none>
I20260924 13:43:09.331974 29600 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:09.332043 29600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:09.332237 29600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:43:09.333292 29600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "a84c175d1fe24841b8804dfffee58bc3"
format_stamp: "Formatted at 2026-09-24 13:43:09 on dist-test-slave-tfm9"
server_key: "02b41446c44fe4e9673110814cd13169"
server_key_iv: "77ed781c137fbf49cb83e0ca63787a20"
server_key_version: "encryptionkey@0"
I20260924 13:43:09.333642 29600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "a84c175d1fe24841b8804dfffee58bc3"
format_stamp: "Formatted at 2026-09-24 13:43:09 on dist-test-slave-tfm9"
server_key: "02b41446c44fe4e9673110814cd13169"
server_key_iv: "77ed781c137fbf49cb83e0ca63787a20"
server_key_version: "encryptionkey@0"
I20260924 13:43:09.334985 29600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:43:09.335685 29614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:09.335865 29600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:43:09.335959 29600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "a84c175d1fe24841b8804dfffee58bc3"
format_stamp: "Formatted at 2026-09-24 13:43:09 on dist-test-slave-tfm9"
server_key: "02b41446c44fe4e9673110814cd13169"
server_key_iv: "77ed781c137fbf49cb83e0ca63787a20"
server_key_version: "encryptionkey@0"
I20260924 13:43:09.336043 29600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:09.339406 29600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:09.339664 29600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:09.339810 29600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:09.340021 29600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:43:09.340310 29600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:43:09.340365 29600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:09.340404 29600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:43:09.340431 29600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:09.346267 29600 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:33719
I20260924 13:43:09.346303 29728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:33719 every 8 connection(s)
I20260924 13:43:09.346623 29600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:43:09.346902 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 29600
I20260924 13:43:09.346993 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:43:09.347200 19671 external_mini_cluster.cc:1428] Setting key 289e3e6cee65cec34d1b3aab66fb1b43
I20260924 13:43:09.348229 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33313
--builtin_ntp_servers=127.19.53.212:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 13:43:09.350162 29447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:09.347361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:45121 (local address 127.19.53.254:33313)
0924 13:43:09.347565 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:09.347567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:09.347751 (+ 184us) server_negotiation.cc:407] Connection header received
0924 13:43:09.348515 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:09.348516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:09.348521 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:09.348567 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:09.349905 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:09.349907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:09.349907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:43:09.349929 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:09.349943 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:43:09.349947 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:09.349999 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:09.350075 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 13:43:09.350089 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260924 13:43:09.350358 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:09.347341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33313 (local address 127.19.53.194:45121)
0924 13:43:09.347618 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:09.347660 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:09.348326 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:09.348645 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:09.348650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:09.348661 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:09.349741 (+ 1080us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:43:09.349742 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:09.349784 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:43:09.349823 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:43:09.349996 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:43:09.350002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:43:09.350005 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:43:09.350082 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 13:43:09.350141 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
I20260924 13:43:09.350795 29729 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33313
I20260924 13:43:09.350886 29729 heartbeater.cc:464] Registering TS with master...
I20260924 13:43:09.351017 29729 heartbeater.cc:511] Master 127.19.53.254:33313 requested a full tablet report, sending...
I20260924 13:43:09.351347 29408 ts_manager.cc:194] Registered new tserver with Master: a84c175d1fe24841b8804dfffee58bc3 (127.19.53.194:33719)
W20260924 13:43:09.429869 29732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:09.430130 29732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:09.430152 29732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:09.431313 29732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:09.431352 29732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:09.431377 29732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:43:09.432685 29732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:09.432933 29732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:09.433229 29732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:09.438166 29740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:09.438189 29738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:09.438187 29737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:09.438577 29732 server_base.cc:1034] running on GCE node
I20260924 13:43:09.438856 29732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:09.439116 29732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:09.440306 29732 hybrid_clock.cc:648] HybridClock initialized: now 1790257389440293 us; error 32 us; skew 500 ppm
I20260924 13:43:09.441664 29732 webserver.cc:460] Webserver started at http://127.19.53.195:32813/ using document root <none> and password file <none>
I20260924 13:43:09.441975 29732 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:09.442050 29732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:09.442190 29732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:43:09.443239 29732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "f6e5e07b6068452d86332e22de076e2c"
format_stamp: "Formatted at 2026-09-24 13:43:09 on dist-test-slave-tfm9"
server_key: "e5a7485235078390a2f8b421924e1807"
server_key_iv: "4f61c9d4c338f02ba30ecdd1c72d236c"
server_key_version: "encryptionkey@0"
I20260924 13:43:09.443617 29732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "f6e5e07b6068452d86332e22de076e2c"
format_stamp: "Formatted at 2026-09-24 13:43:09 on dist-test-slave-tfm9"
server_key: "e5a7485235078390a2f8b421924e1807"
server_key_iv: "4f61c9d4c338f02ba30ecdd1c72d236c"
server_key_version: "encryptionkey@0"
I20260924 13:43:09.445034 29732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:43:09.445740 29746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:09.445896 29732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:43:09.446038 29732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "f6e5e07b6068452d86332e22de076e2c"
format_stamp: "Formatted at 2026-09-24 13:43:09 on dist-test-slave-tfm9"
server_key: "e5a7485235078390a2f8b421924e1807"
server_key_iv: "4f61c9d4c338f02ba30ecdd1c72d236c"
server_key_version: "encryptionkey@0"
I20260924 13:43:09.446120 29732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:09.449373 29732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:09.449625 29732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:09.449769 29732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:09.450021 29732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:43:09.450328 29732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:43:09.450382 29732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:09.450421 29732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:43:09.450448 29732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:09.456166 29732 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:39679
I20260924 13:43:09.456202 29860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:39679 every 8 connection(s)
I20260924 13:43:09.456516 29732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:43:09.459491 29447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:09.457249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:38611 (local address 127.19.53.254:33313)
0924 13:43:09.457294 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:09.457296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:09.457497 (+ 201us) server_negotiation.cc:407] Connection header received
0924 13:43:09.458033 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:09.458035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:09.458042 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:09.458092 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:09.459211 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:09.459212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:09.459213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:43:09.459217 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:09.459232 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:43:09.459236 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:09.459288 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:09.459405 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 13:43:09.459421 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:43:09.459686 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:09.457211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33313 (local address 127.19.53.195:38611)
0924 13:43:09.457384 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:09.457414 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:09.457877 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:09.458157 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:09.458161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:09.458171 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:09.459055 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:43:09.459056 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:09.459082 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:43:09.459107 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:43:09.459296 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:43:09.459301 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:43:09.459304 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:43:09.459416 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 13:43:09.459474 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
I20260924 13:43:09.460146 29861 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33313
I20260924 13:43:09.460268 29861 heartbeater.cc:464] Registering TS with master...
I20260924 13:43:09.460434 29861 heartbeater.cc:511] Master 127.19.53.254:33313 requested a full tablet report, sending...
I20260924 13:43:09.460779 29408 ts_manager.cc:194] Registered new tserver with Master: f6e5e07b6068452d86332e22de076e2c (127.19.53.195:39679)
I20260924 13:43:09.463966 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 29732
I20260924 13:43:09.464037 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:43:09.464226 19671 external_mini_cluster.cc:1428] Setting key cf8d62781f2da9ba88d29e0bb864322d
I20260924 13:43:09.465085 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 13:43:09.466948 29447 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:43:09.466433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.19.53.254:33313)
0924 13:43:09.466512 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:09.466514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:09.466616 (+ 102us) server_negotiation.cc:407] Connection header received
0924 13:43:09.466684 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:09.466685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:09.466690 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:09.466746 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:09.466866 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48834: BlockingRecv error: recv got EOF from 127.0.0.1:48834 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 13:43:09.467038 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 29468
I20260924 13:43:09.471976 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 29600
I20260924 13:43:09.476248 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 29732
I20260924 13:43:09.480676 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 29376
2026-09-24T13:43:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (524 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:43:09 dist-test-slave-tfm9 krb5kdc[29871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:43:09 dist-test-slave-tfm9 krb5kdc[29871](info): set up 2 sockets
Sep 24 13:43:09 dist-test-slave-tfm9 krb5kdc[29871](info): commencing operation
krb5kdc: starting...
W20260924 13:43:10.512424 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:43:10 dist-test-slave-tfm9 krb5kdc[29871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:43:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:43:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:43:10.628419 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:43667
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:43667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:43:10.709031 29885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:10.709223 29885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:10.709244 29885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:10.710428 29885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:43:10.710471 29885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:10.710486 29885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:10.710500 29885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:43:10.710511 29885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:43:10.711966 29885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:43667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:43667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:10.712191 29885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:10.712471 29885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:10.717355 29891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:10.717366 29893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:10.717365 29890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:10.717705 29885 server_base.cc:1034] running on GCE node
I20260924 13:43:10.718015 29885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:10.718320 29885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:10.719482 29885 hybrid_clock.cc:648] HybridClock initialized: now 1790257390719461 us; error 36 us; skew 500 ppm
Sep 24 13:43:10 dist-test-slave-tfm9 krb5kdc[29871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:43:10.722146 29885 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:43:10.722672 29885 webserver.cc:460] Webserver started at http://127.19.53.254:36391/ using document root <none> and password file <none>
I20260924 13:43:10.722916 29885 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:10.722983 29885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:10.723114 29885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:43:10.724126 29885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "8c21a605549b4f3692cf3e75dcbc61ef"
format_stamp: "Formatted at 2026-09-24 13:43:10 on dist-test-slave-tfm9"
server_key: "54c1947292cd0c9b708d8055ec9b8711"
server_key_iv: "4040c7cbec68092ede1b35d2641d049b"
server_key_version: "encryptionkey@0"
I20260924 13:43:10.724483 29885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "8c21a605549b4f3692cf3e75dcbc61ef"
format_stamp: "Formatted at 2026-09-24 13:43:10 on dist-test-slave-tfm9"
server_key: "54c1947292cd0c9b708d8055ec9b8711"
server_key_iv: "4040c7cbec68092ede1b35d2641d049b"
server_key_version: "encryptionkey@0"
I20260924 13:43:10.725741 29885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:43:10.726565 29900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:10.726734 29885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:43:10.726841 29885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "8c21a605549b4f3692cf3e75dcbc61ef"
format_stamp: "Formatted at 2026-09-24 13:43:10 on dist-test-slave-tfm9"
server_key: "54c1947292cd0c9b708d8055ec9b8711"
server_key_iv: "4040c7cbec68092ede1b35d2641d049b"
server_key_version: "encryptionkey@0"
I20260924 13:43:10.726927 29885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:10.741328 29885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:10.743480 29885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:10.743641 29885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:10.747594 29885 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:43667
I20260924 13:43:10.747630 29953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:43667 every 8 connection(s)
I20260924 13:43:10.747967 29885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:43:10.749028 29954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:43:10.751552 29954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef: Bootstrap starting.
I20260924 13:43:10.752130 29954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef: Neither blocks nor log segments found. Creating new log.
I20260924 13:43:10.752400 29954 log.cc:824] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef: Log is configured to *not* fsync() on all Append() calls
I20260924 13:43:10.753324 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 29885
I20260924 13:43:10.753391 29954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef: No bootstrap required, opened a new log
I20260924 13:43:10.753449 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:43:10.753661 19671 external_mini_cluster.cc:1428] Setting key 7eebbe58b8e726b15aa7aa7fc6b1ad3b
I20260924 13:43:10.755417 29954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c21a605549b4f3692cf3e75dcbc61ef" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43667 } }
I20260924 13:43:10.755575 29954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:43:10.755677 29954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c21a605549b4f3692cf3e75dcbc61ef, State: Initialized, Role: FOLLOWER
I20260924 13:43:10.755851 29954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c21a605549b4f3692cf3e75dcbc61ef" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43667 } }
I20260924 13:43:10.755949 29954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:43:10.755983 29954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:43:10.756040 29954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:43:10.757025 29954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c21a605549b4f3692cf3e75dcbc61ef" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43667 } }
I20260924 13:43:10.757146 29954 leader_election.cc:302] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c21a605549b4f3692cf3e75dcbc61ef; no voters:
I20260924 13:43:10.757321 29954 leader_election.cc:288] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:43:10.757385 29959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:43:10.757584 29954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:43:10.757611 29959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [term 1 LEADER]: Becoming Leader. State: Replica: 8c21a605549b4f3692cf3e75dcbc61ef, State: Running, Role: LEADER
I20260924 13:43:10.757864 29959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c21a605549b4f3692cf3e75dcbc61ef" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43667 } }
I20260924 13:43:10.758417 29960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c21a605549b4f3692cf3e75dcbc61ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c21a605549b4f3692cf3e75dcbc61ef" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43667 } } }
I20260924 13:43:10.758512 29960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [sys.catalog]: This master's current role is: LEADER
I20260924 13:43:10.759017 29961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c21a605549b4f3692cf3e75dcbc61ef. Latest consensus state: current_term: 1 leader_uuid: "8c21a605549b4f3692cf3e75dcbc61ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c21a605549b4f3692cf3e75dcbc61ef" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43667 } } }
I20260924 13:43:10.759222 29968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:43:10.759351 29961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef [sys.catalog]: This master's current role is: LEADER
I20260924 13:43:10.759976 29968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:43:10.762048 29968 catalog_manager.cc:1347] Generated new cluster ID: 0f319e50001d4053aa4158ef80353082
I20260924 13:43:10.762101 29968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:43:10 dist-test-slave-tfm9 krb5kdc[29871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:43:10.766844 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:10.754307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.19.53.254:43667)
0924 13:43:10.754469 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:10.754476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:10.754501 (+ 25us) server_negotiation.cc:407] Connection header received
0924 13:43:10.755406 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:10.755413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:10.755467 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:10.755624 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:10.756415 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:10.757750 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:10.758978 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:10.759649 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:10.759764 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:10.762877 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:10.762881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:10.762882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:43:10.762895 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:10.764906 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:10.765366 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:10.765369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:10.765370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:10.765506 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:10.765763 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:10.765764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:10.765764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:10.766122 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:10.766340 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:10.766416 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 13:43:10.766682 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":89,"threads_started":1}
I20260924 13:43:10.777449 29968 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:43:10.778045 29968 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:43:10.782168 29968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef: Generated new TSK 0
I20260924 13:43:10.782411 29968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:43:10.810175 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:43667
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:43:10.913208 29980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:10.913439 29980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:10.913479 29980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:10.914819 29980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:10.914887 29980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:10.914932 29980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:43:10.916361 29980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:43667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:10.916620 29980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:10.916858 29980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:10.922495 29985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:10.922523 29986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:10.922508 29988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:10.922819 29980 server_base.cc:1034] running on GCE node
I20260924 13:43:10.923026 29980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:10.923226 29980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:10.924396 29980 hybrid_clock.cc:648] HybridClock initialized: now 1790257390924377 us; error 35 us; skew 500 ppm
Sep 24 13:43:10 dist-test-slave-tfm9 krb5kdc[29871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:43:10.926949 29980 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:43:10.927428 29980 webserver.cc:460] Webserver started at http://127.19.53.193:40693/ using document root <none> and password file <none>
I20260924 13:43:10.927683 29980 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:10.927748 29980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:10.927891 29980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:43:10.928954 29980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b"
format_stamp: "Formatted at 2026-09-24 13:43:10 on dist-test-slave-tfm9"
server_key: "8c0bb4083904c55f4c235dd0d89ad934"
server_key_iv: "03b0190ccde370452989c5e7db354232"
server_key_version: "encryptionkey@0"
I20260924 13:43:10.929352 29980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b"
format_stamp: "Formatted at 2026-09-24 13:43:10 on dist-test-slave-tfm9"
server_key: "8c0bb4083904c55f4c235dd0d89ad934"
server_key_iv: "03b0190ccde370452989c5e7db354232"
server_key_version: "encryptionkey@0"
I20260924 13:43:10.930800 29980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:43:10.931571 29995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:10.931732 29980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:10.931826 29980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b"
format_stamp: "Formatted at 2026-09-24 13:43:10 on dist-test-slave-tfm9"
server_key: "8c0bb4083904c55f4c235dd0d89ad934"
server_key_iv: "03b0190ccde370452989c5e7db354232"
server_key_version: "encryptionkey@0"
I20260924 13:43:10.931908 29980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:10.948238 29980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:10.950191 29980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:10.950364 29980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:10.950587 29980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:43:10.950887 29980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:43:10.950941 29980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:10.950980 29980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:43:10.951009 29980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:10.956761 29980 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:35263
I20260924 13:43:10.956807 30109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:35263 every 8 connection(s)
I20260924 13:43:10.957126 29980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:43:10 dist-test-slave-tfm9 krb5kdc[29871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:43:10.965809 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 29980
I20260924 13:43:10.965911 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:43:10.966136 19671 external_mini_cluster.cc:1428] Setting key a6219e22132eef75660977faf2b0f31e
I20260924 13:43:10.966153 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:10.957836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:48373 (local address 127.19.53.254:43667)
0924 13:43:10.957945 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:10.957948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:10.958146 (+ 198us) server_negotiation.cc:407] Connection header received
0924 13:43:10.958685 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:10.958690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:10.958701 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:10.958785 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:10.959704 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:10.960216 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:10.961155 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:10.961392 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:10.961447 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:10.963463 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:10.963464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:10.963465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:43:10.963511 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:10.964730 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:10.965157 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:10.965158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:10.965158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:10.965213 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:10.965457 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:10.965458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:10.965458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:10.965566 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:10.965643 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:10.965966 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 13:43:10.966064 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:43:10.966288 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:10.957857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:43667 (local address 127.19.53.193:48373)
0924 13:43:10.958030 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:10.958066 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:10.958557 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:10.958889 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:10.958892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:10.959435 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:10.959622 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:10.959626 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:10.960293 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:10.960294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:10.961076 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:10.961078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:10.961466 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:10.961467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:10.961561 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:10.962062 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:10.962070 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:10.963370 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:43:10.964847 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:43:10.964854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:43:10.964856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:43:10.965072 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:43:10.965280 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:43:10.965281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:43:10.965282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:43:10.965385 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:43:10.965649 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:43:10.965656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:43:10.965791 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 13:43:10.965894 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 13:43:10.966073 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260924 13:43:10.966894 30110 heartbeater.cc:347] Connected to a master server at 127.19.53.254:43667
I20260924 13:43:10.966993 30110 heartbeater.cc:464] Registering TS with master...
I20260924 13:43:10.967163 30110 heartbeater.cc:511] Master 127.19.53.254:43667 requested a full tablet report, sending...
I20260924 13:43:10.967700 29917 ts_manager.cc:194] Registered new tserver with Master: 908c3c70c92e4e1bbd5a0d1bbbc2754b (127.19.53.193:35263)
I20260924 13:43:10.968586 29917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:48373
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:43:10.990835 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:43667
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:43:11.070611 30115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:11.070804 30115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:11.070835 30115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:11.071959 30115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:11.072018 30115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:11.072043 30115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:43:11.073478 30115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:43667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:11.073761 30115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:11.074023 30115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:11.079020 30120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:11.079056 30121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:11.079020 30123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:11.079262 30115 server_base.cc:1034] running on GCE node
I20260924 13:43:11.079473 30115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:11.079702 30115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:11.080864 30115 hybrid_clock.cc:648] HybridClock initialized: now 1790257391080850 us; error 42 us; skew 500 ppm
Sep 24 13:43:11 dist-test-slave-tfm9 krb5kdc[29871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257391, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:43:11.083338 30115 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:43:11.083694 30115 webserver.cc:460] Webserver started at http://127.19.53.194:43759/ using document root <none> and password file <none>
I20260924 13:43:11.083899 30115 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:11.083937 30115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:11.084022 30115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:43:11.084954 30115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "b7db83a0716148eaa97a6aaf601c4e73"
format_stamp: "Formatted at 2026-09-24 13:43:11 on dist-test-slave-tfm9"
server_key: "c5831fd65e3fb4964797df474f870532"
server_key_iv: "e8a628ff344669e38a811c1abbd199cc"
server_key_version: "encryptionkey@0"
I20260924 13:43:11.085263 30115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "b7db83a0716148eaa97a6aaf601c4e73"
format_stamp: "Formatted at 2026-09-24 13:43:11 on dist-test-slave-tfm9"
server_key: "c5831fd65e3fb4964797df474f870532"
server_key_iv: "e8a628ff344669e38a811c1abbd199cc"
server_key_version: "encryptionkey@0"
I20260924 13:43:11.086568 30115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:43:11.087242 30130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:11.087450 30115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:43:11.087525 30115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "b7db83a0716148eaa97a6aaf601c4e73"
format_stamp: "Formatted at 2026-09-24 13:43:11 on dist-test-slave-tfm9"
server_key: "c5831fd65e3fb4964797df474f870532"
server_key_iv: "e8a628ff344669e38a811c1abbd199cc"
server_key_version: "encryptionkey@0"
I20260924 13:43:11.087576 30115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:11.116366 30115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:11.118778 30115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:11.118889 30115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:11.119064 30115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:43:11.119380 30115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:43:11.119414 30115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:11.119457 30115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:43:11.119484 30115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:11.125387 30115 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:41941
I20260924 13:43:11.125452 30244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:41941 every 8 connection(s)
I20260924 13:43:11.125752 30115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:43:11 dist-test-slave-tfm9 krb5kdc[29871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257391, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:43:11.135210 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.126500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:50561 (local address 127.19.53.254:43667)
0924 13:43:11.126551 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:11.126555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:11.126785 (+ 230us) server_negotiation.cc:407] Connection header received
0924 13:43:11.127261 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:11.127263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:11.127273 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:11.127327 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:11.128248 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.128730 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.129826 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.130080 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.130135 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.132262 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:11.132263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:11.132263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:43:11.132271 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:11.133733 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:11.134191 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:11.134193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:11.134193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:11.134259 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:11.134555 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:11.134556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:11.134557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:11.134673 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:11.134747 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:11.135027 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 13:43:11.135108 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:43:11.135375 30247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.126490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:43667 (local address 127.19.53.194:50561)
0924 13:43:11.126661 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:11.126696 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:11.127179 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:11.127381 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:11.127385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:11.127975 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:11.128171 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.128175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.128817 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.128818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.129746 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.129748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.130148 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.130149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.130294 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.130736 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:11.130751 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:11.132165 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:43:11.133854 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:43:11.133860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:43:11.133862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:43:11.134115 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:43:11.134327 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:43:11.134327 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:43:11.134328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:43:11.134439 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:43:11.134751 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:43:11.134757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:43:11.134889 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 13:43:11.134994 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 13:43:11.135143 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260924 13:43:11.135622 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 30115
I20260924 13:43:11.135720 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:43:11.135913 30245 heartbeater.cc:347] Connected to a master server at 127.19.53.254:43667
I20260924 13:43:11.135949 19671 external_mini_cluster.cc:1428] Setting key efa935fc74159ebc6dbdf56d65ad2f18
I20260924 13:43:11.136003 30245 heartbeater.cc:464] Registering TS with master...
I20260924 13:43:11.136189 30245 heartbeater.cc:511] Master 127.19.53.254:43667 requested a full tablet report, sending...
I20260924 13:43:11.136562 29917 ts_manager.cc:194] Registered new tserver with Master: b7db83a0716148eaa97a6aaf601c4e73 (127.19.53.194:41941)
I20260924 13:43:11.136969 29917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:50561
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:43:11.158631 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:43667
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:43:11.238178 30250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:11.238363 30250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:11.238385 30250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:11.239482 30250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:11.239521 30250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:11.239571 30250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:43:11.241017 30250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:43667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:11.241261 30250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:11.241511 30250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:11.246429 30255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:11.246450 30256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:11.246459 30258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:11.246676 30250 server_base.cc:1034] running on GCE node
I20260924 13:43:11.246877 30250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:11.247072 30250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:11.248206 30250 hybrid_clock.cc:648] HybridClock initialized: now 1790257391248196 us; error 31 us; skew 500 ppm
Sep 24 13:43:11 dist-test-slave-tfm9 krb5kdc[29871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257391, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:43:11.250886 30250 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:43:11.251423 30250 webserver.cc:460] Webserver started at http://127.19.53.195:41761/ using document root <none> and password file <none>
I20260924 13:43:11.251679 30250 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:11.251755 30250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:11.251904 30250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:43:11.252969 30250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "3c99ca064bfb457caabbffb7245671bc"
format_stamp: "Formatted at 2026-09-24 13:43:11 on dist-test-slave-tfm9"
server_key: "bcea21fb3708ecf851b23f26bf609f8e"
server_key_iv: "196fbc52d4a545d0b2cb9fe08aed24d1"
server_key_version: "encryptionkey@0"
I20260924 13:43:11.253364 30250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "3c99ca064bfb457caabbffb7245671bc"
format_stamp: "Formatted at 2026-09-24 13:43:11 on dist-test-slave-tfm9"
server_key: "bcea21fb3708ecf851b23f26bf609f8e"
server_key_iv: "196fbc52d4a545d0b2cb9fe08aed24d1"
server_key_version: "encryptionkey@0"
I20260924 13:43:11.254837 30250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:43:11.255633 30265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:11.255815 30250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:43:11.255916 30250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "3c99ca064bfb457caabbffb7245671bc"
format_stamp: "Formatted at 2026-09-24 13:43:11 on dist-test-slave-tfm9"
server_key: "bcea21fb3708ecf851b23f26bf609f8e"
server_key_iv: "196fbc52d4a545d0b2cb9fe08aed24d1"
server_key_version: "encryptionkey@0"
I20260924 13:43:11.256000 30250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:11.269232 30250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:11.271162 30250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:11.271368 30250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:11.271580 30250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:43:11.271863 30250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:43:11.271914 30250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:11.271953 30250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:43:11.271978 30250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:11.277724 30250 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:42067
I20260924 13:43:11.277772 30379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:42067 every 8 connection(s)
I20260924 13:43:11.278136 30250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:43:11.283116 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 30250
I20260924 13:43:11.283213 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:43:11.283389 19671 external_mini_cluster.cc:1428] Setting key 96c00bd11d22c6d27b98150c954ab5a4
Sep 24 13:43:11 dist-test-slave-tfm9 krb5kdc[29871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257391, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:43:11.287137 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.278873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:38015 (local address 127.19.53.254:43667)
0924 13:43:11.278938 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:11.278940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:11.279160 (+ 220us) server_negotiation.cc:407] Connection header received
0924 13:43:11.279657 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:11.279658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:11.279668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:11.279724 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:11.280597 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.281075 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.282055 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.282392 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.282449 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.284861 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:11.284863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:11.284863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:43:11.284870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:11.286069 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:11.286410 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:11.286411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:11.286411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:11.286460 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:11.286660 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:11.286661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:11.286661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:11.286755 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:11.286806 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:11.287012 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 13:43:11.287075 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 13:43:11.287428 30382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.278848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:43667 (local address 127.19.53.195:38015)
0924 13:43:11.279036 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:11.279075 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:11.279574 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:11.279777 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:11.279780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:11.280337 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:11.280522 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.280528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.281150 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.281151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.281964 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.281966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.282470 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.282471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.282571 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.283101 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:11.283109 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:11.284776 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:43:11.286133 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:43:11.286138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:43:11.286140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:43:11.286348 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:43:11.286516 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:43:11.286516 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:43:11.286517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:43:11.286609 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:43:11.286807 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:43:11.286811 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:43:11.286920 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 13:43:11.286999 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 13:43:11.287110 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":120,"threads_started":1}
I20260924 13:43:11.287844 30380 heartbeater.cc:347] Connected to a master server at 127.19.53.254:43667
I20260924 13:43:11.287964 30380 heartbeater.cc:464] Registering TS with master...
I20260924 13:43:11.288143 30380 heartbeater.cc:511] Master 127.19.53.254:43667 requested a full tablet report, sending...
I20260924 13:43:11.288499 29917 ts_manager.cc:194] Registered new tserver with Master: 3c99ca064bfb457caabbffb7245671bc (127.19.53.195:42067)
I20260924 13:43:11.288862 29917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:38015
I20260924 13:43:11.294899 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:43:11.301512 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.296161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.19.53.254:43667)
0924 13:43:11.296221 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:11.296224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:11.296334 (+ 110us) server_negotiation.cc:407] Connection header received
0924 13:43:11.296377 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:11.296378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:11.296386 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:11.296436 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:11.296946 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.297389 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.298081 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.298245 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.298298 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.299408 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:11.299409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:11.299410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:43:11.299415 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:11.300557 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:11.300857 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:11.300857 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:11.300857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:11.300905 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:11.301032 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:11.301032 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:11.301032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:11.301170 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:11.301240 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:11.301351 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 13:43:11.301406 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:43:11.303145 29917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:43:11.304662 29917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:43:11.310933 30396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.307331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.19.53.193:35263)
0924 13:43:11.307486 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:11.307491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:11.307499 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:43:11.307535 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:11.307536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:11.307550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:11.307740 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:43:11.307816 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.308889 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.309994 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.310525 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.310616 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.310675 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:11.310793 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 13:43:11.310843 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":99,"threads_started":1}
I20260924 13:43:11.310936 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.307119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:35263 (local address 127.0.0.1:41806)
0924 13:43:11.307312 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:11.307321 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:11.307358 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:11.307637 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:11.307637 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:11.307639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:43:11.307675 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.307676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.309004 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.309005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.309864 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.309866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.310616 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.310617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.310694 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.310707 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 13:43:11.310779 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 13:43:11.310832 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":59,"threads_started":1}
I20260924 13:43:11.310952 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.306947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:42067 (local address 127.0.0.1:33892)
0924 13:43:11.307097 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:11.307135 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:11.307203 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:11.307542 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:11.307545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:11.307547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:43:11.307622 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.307623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.308583 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.308584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.309769 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.309772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.310695 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.310696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.310757 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.310761 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:43:11.310805 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 13:43:11.310854 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":41,"threads_started":1}
I20260924 13:43:11.311187 30393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.307013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33892 (local address 127.19.53.195:42067)
0924 13:43:11.307363 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:11.307367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:11.307377 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:43:11.307415 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:11.307418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:11.307433 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:11.307701 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:43:11.307766 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.308475 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.309845 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.310592 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.310988 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.311047 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:11.311071 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 13:43:11.311109 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":71,"threads_started":1}
I20260924 13:43:11.311275 30394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.307054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.19.53.194:41941)
0924 13:43:11.307214 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:11.307220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:11.307382 (+ 162us) server_negotiation.cc:407] Connection header received
0924 13:43:11.307427 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:11.307429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:11.307445 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:11.307516 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:43:11.307718 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.308850 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.310122 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.310816 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.310876 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.310930 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:11.311152 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 13:43:11.311195 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260924 13:43:11.311440 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.306952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:41941 (local address 127.0.0.1:58324)
0924 13:43:11.307219 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:11.307230 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:11.307314 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:11.307596 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:11.307597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:11.307599 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:43:11.307645 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.307647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.308963 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.308964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.310037 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.310039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.310961 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.310962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.311025 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.311028 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:43:11.311078 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:43:11.311109 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":72,"threads_started":1}
I20260924 13:43:11.312024 30314 tablet_service.cc:1467] Processing CreateTablet for tablet f6b86340f98c459fa7396975b861553c (DEFAULT_TABLE table=test-table [id=c79f3c66985040b8acae51504059534f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:43:11.312045 30179 tablet_service.cc:1467] Processing CreateTablet for tablet f6b86340f98c459fa7396975b861553c (DEFAULT_TABLE table=test-table [id=c79f3c66985040b8acae51504059534f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:43:11.312417 30314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b86340f98c459fa7396975b861553c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:43:11.312417 30179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b86340f98c459fa7396975b861553c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:43:11.314507 30044 tablet_service.cc:1467] Processing CreateTablet for tablet f6b86340f98c459fa7396975b861553c (DEFAULT_TABLE table=test-table [id=c79f3c66985040b8acae51504059534f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:43:11.314859 30044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b86340f98c459fa7396975b861553c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:43:11.315744 30398 tablet_bootstrap.cc:492] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc: Bootstrap starting.
I20260924 13:43:11.316385 30398 tablet_bootstrap.cc:654] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc: Neither blocks nor log segments found. Creating new log.
I20260924 13:43:11.316913 30398 log.cc:824] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc: Log is configured to *not* fsync() on all Append() calls
I20260924 13:43:11.317231 30397 tablet_bootstrap.cc:492] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73: Bootstrap starting.
I20260924 13:43:11.317595 30398 tablet_bootstrap.cc:492] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc: No bootstrap required, opened a new log
I20260924 13:43:11.317637 30398 ts_tablet_manager.cc:1397] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:43:11.317943 30400 tablet_bootstrap.cc:492] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b: Bootstrap starting.
I20260924 13:43:11.317849 30397 tablet_bootstrap.cc:654] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73: Neither blocks nor log segments found. Creating new log.
I20260924 13:43:11.318625 30400 tablet_bootstrap.cc:654] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b: Neither blocks nor log segments found. Creating new log.
I20260924 13:43:11.318905 30397 log.cc:824] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73: Log is configured to *not* fsync() on all Append() calls
I20260924 13:43:11.319162 30400 log.cc:824] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:43:11.319585 30397 tablet_bootstrap.cc:492] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73: No bootstrap required, opened a new log
I20260924 13:43:11.319664 30397 ts_tablet_manager.cc:1397] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:43:11.319828 30400 tablet_bootstrap.cc:492] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b: No bootstrap required, opened a new log
I20260924 13:43:11.319893 30400 ts_tablet_manager.cc:1397] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:43:11.319793 30398 raft_consensus.cc:348] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c99ca064bfb457caabbffb7245671bc" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 42067 } } peers { permanent_uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35263 } } peers { permanent_uuid: "b7db83a0716148eaa97a6aaf601c4e73" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 41941 } }
I20260924 13:43:11.319996 30398 raft_consensus.cc:374] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:43:11.320070 30398 raft_consensus.cc:729] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c99ca064bfb457caabbffb7245671bc, State: Initialized, Role: FOLLOWER
I20260924 13:43:11.320218 30398 consensus_queue.cc:260] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c99ca064bfb457caabbffb7245671bc" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 42067 } } peers { permanent_uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35263 } } peers { permanent_uuid: "b7db83a0716148eaa97a6aaf601c4e73" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 41941 } }
I20260924 13:43:11.320515 30380 heartbeater.cc:503] Master 127.19.53.254:43667 was elected leader, sending a full tablet report...
I20260924 13:43:11.320752 30398 ts_tablet_manager.cc:1428] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:43:11.321643 30400 raft_consensus.cc:348] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c99ca064bfb457caabbffb7245671bc" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 42067 } } peers { permanent_uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35263 } } peers { permanent_uuid: "b7db83a0716148eaa97a6aaf601c4e73" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 41941 } }
I20260924 13:43:11.321710 30397 raft_consensus.cc:348] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c99ca064bfb457caabbffb7245671bc" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 42067 } } peers { permanent_uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35263 } } peers { permanent_uuid: "b7db83a0716148eaa97a6aaf601c4e73" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 41941 } }
I20260924 13:43:11.321830 30400 raft_consensus.cc:374] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:43:11.321866 30397 raft_consensus.cc:374] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:43:11.321879 30400 raft_consensus.cc:729] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 908c3c70c92e4e1bbd5a0d1bbbc2754b, State: Initialized, Role: FOLLOWER
I20260924 13:43:11.321938 30397 raft_consensus.cc:729] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7db83a0716148eaa97a6aaf601c4e73, State: Initialized, Role: FOLLOWER
I20260924 13:43:11.322016 30400 consensus_queue.cc:260] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c99ca064bfb457caabbffb7245671bc" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 42067 } } peers { permanent_uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35263 } } peers { permanent_uuid: "b7db83a0716148eaa97a6aaf601c4e73" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 41941 } }
I20260924 13:43:11.322059 30397 consensus_queue.cc:260] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c99ca064bfb457caabbffb7245671bc" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 42067 } } peers { permanent_uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35263 } } peers { permanent_uuid: "b7db83a0716148eaa97a6aaf601c4e73" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 41941 } }
I20260924 13:43:11.322306 30400 ts_tablet_manager.cc:1428] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 13:43:11.322335 30110 heartbeater.cc:503] Master 127.19.53.254:43667 was elected leader, sending a full tablet report...
I20260924 13:43:11.322326 30245 heartbeater.cc:503] Master 127.19.53.254:43667 was elected leader, sending a full tablet report...
I20260924 13:43:11.322477 30397 ts_tablet_manager.cc:1428] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 13:43:11.358404 30405 raft_consensus.cc:482] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:43:11.358553 30405 raft_consensus.cc:504] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c99ca064bfb457caabbffb7245671bc" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 42067 } } peers { permanent_uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35263 } } peers { permanent_uuid: "b7db83a0716148eaa97a6aaf601c4e73" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 41941 } }
I20260924 13:43:11.359050 30405 leader_election.cc:288] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c99ca064bfb457caabbffb7245671bc (127.19.53.195:42067), 908c3c70c92e4e1bbd5a0d1bbbc2754b (127.19.53.193:35263)
I20260924 13:43:11.362948 30396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.359490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:58387 (local address 127.19.53.193:35263)
0924 13:43:11.359537 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:11.359539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:11.359685 (+ 146us) server_negotiation.cc:407] Connection header received
0924 13:43:11.359749 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:11.359750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:11.359762 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:11.359821 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:43:11.360021 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.360711 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.361982 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.362572 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.362660 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.362669 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:11.362836 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 13:43:11.362875 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:43:11.362948 30247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.359265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:42067 (local address 127.19.53.194:40471)
0924 13:43:11.359346 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:11.359361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:11.359417 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:11.359636 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:11.359638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:11.359640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:43:11.359727 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.359729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.360659 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.360659 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.361868 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.361870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.362668 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.362669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.362758 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.362760 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:43:11.362813 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 13:43:11.362858 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":3}
I20260924 13:43:11.362948 30393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.359407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:40471 (local address 127.19.53.195:42067)
0924 13:43:11.359487 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:11.359489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:11.359500 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:43:11.359532 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:11.359533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:11.359544 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:11.359618 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:43:11.359779 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.360575 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.361982 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.362572 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.362660 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.362669 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:11.362836 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 13:43:11.362875 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 13:43:11.362955 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.359398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:35263 (local address 127.19.53.194:58387)
0924 13:43:11.359574 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:11.359584 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:11.359642 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:11.359868 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:11.359869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:11.359872 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:43:11.359932 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.359933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.360789 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.360790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.361868 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:11.361870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.362668 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.362669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:11.362750 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.362755 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:43:11.362823 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 13:43:11.362858 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":5,"thread_start_us":114,"threads_started":1}
I20260924 13:43:11.363363 30334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b86340f98c459fa7396975b861553c" candidate_uuid: "b7db83a0716148eaa97a6aaf601c4e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c99ca064bfb457caabbffb7245671bc" is_pre_election: true
I20260924 13:43:11.363363 30064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b86340f98c459fa7396975b861553c" candidate_uuid: "b7db83a0716148eaa97a6aaf601c4e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b" is_pre_election: true
I20260924 13:43:11.363543 30334 raft_consensus.cc:2454] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7db83a0716148eaa97a6aaf601c4e73 in term 0.
I20260924 13:43:11.363555 30064 raft_consensus.cc:2454] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7db83a0716148eaa97a6aaf601c4e73 in term 0.
I20260924 13:43:11.363802 30133 leader_election.cc:302] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 908c3c70c92e4e1bbd5a0d1bbbc2754b, b7db83a0716148eaa97a6aaf601c4e73; no voters:
I20260924 13:43:11.363976 30405 raft_consensus.cc:2781] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:43:11.364058 30405 raft_consensus.cc:482] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:43:11.364092 30405 raft_consensus.cc:3037] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:43:11.364887 30405 raft_consensus.cc:504] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c99ca064bfb457caabbffb7245671bc" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 42067 } } peers { permanent_uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35263 } } peers { permanent_uuid: "b7db83a0716148eaa97a6aaf601c4e73" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 41941 } }
I20260924 13:43:11.365026 30405 leader_election.cc:288] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [CANDIDATE]: Term 1 election: Requested vote from peers 3c99ca064bfb457caabbffb7245671bc (127.19.53.195:42067), 908c3c70c92e4e1bbd5a0d1bbbc2754b (127.19.53.193:35263)
I20260924 13:43:11.365162 30334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b86340f98c459fa7396975b861553c" candidate_uuid: "b7db83a0716148eaa97a6aaf601c4e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c99ca064bfb457caabbffb7245671bc"
I20260924 13:43:11.365216 30064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b86340f98c459fa7396975b861553c" candidate_uuid: "b7db83a0716148eaa97a6aaf601c4e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b"
I20260924 13:43:11.365244 30334 raft_consensus.cc:3037] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:43:11.365274 30064 raft_consensus.cc:3037] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:43:11.366173 30334 raft_consensus.cc:2454] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7db83a0716148eaa97a6aaf601c4e73 in term 1.
I20260924 13:43:11.366173 30064 raft_consensus.cc:2454] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7db83a0716148eaa97a6aaf601c4e73 in term 1.
I20260924 13:43:11.366340 30133 leader_election.cc:302] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 908c3c70c92e4e1bbd5a0d1bbbc2754b, b7db83a0716148eaa97a6aaf601c4e73; no voters:
I20260924 13:43:11.366441 30405 raft_consensus.cc:2781] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:43:11.366655 30405 raft_consensus.cc:686] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [term 1 LEADER]: Becoming Leader. State: Replica: b7db83a0716148eaa97a6aaf601c4e73, State: Running, Role: LEADER
I20260924 13:43:11.366777 30405 consensus_queue.cc:237] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c99ca064bfb457caabbffb7245671bc" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 42067 } } peers { permanent_uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35263 } } peers { permanent_uuid: "b7db83a0716148eaa97a6aaf601c4e73" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 41941 } }
I20260924 13:43:11.367712 29917 catalog_manager.cc:5495] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7db83a0716148eaa97a6aaf601c4e73 (127.19.53.194). New cstate: current_term: 1 leader_uuid: "b7db83a0716148eaa97a6aaf601c4e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c99ca064bfb457caabbffb7245671bc" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 42067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7db83a0716148eaa97a6aaf601c4e73" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 41941 } health_report { overall_health: HEALTHY } } }
W20260924 13:43:11.376329 30246 tablet.cc:2369] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:43:11.383169 30394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.381015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.19.53.194:41941)
0924 13:43:11.381073 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:11.381077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:11.381093 (+ 16us) server_negotiation.cc:407] Connection header received
0924 13:43:11.381121 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:11.381122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:11.381137 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:11.381196 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:43:11.381314 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.381755 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.382511 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:11.382720 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:11.382772 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:11.382899 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:43:11.382988 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:43:11.383046 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:11.383083 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 13:43:11.383099 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 13:43:11.384740 30334 raft_consensus.cc:1261] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc [term 1 FOLLOWER]: Refusing update from remote peer b7db83a0716148eaa97a6aaf601c4e73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:43:11.384781 30064 raft_consensus.cc:1261] T f6b86340f98c459fa7396975b861553c P 908c3c70c92e4e1bbd5a0d1bbbc2754b [term 1 FOLLOWER]: Refusing update from remote peer b7db83a0716148eaa97a6aaf601c4e73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:43:11.385140 30407 consensus_queue.cc:1035] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c99ca064bfb457caabbffb7245671bc" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 42067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:43:11.385268 30405 consensus_queue.cc:1035] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "908c3c70c92e4e1bbd5a0d1bbbc2754b" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:43:11.387969 30411 mvcc.cc:205] Tried to move back new op lower bound from 7332894275109494784 to 7332894275039084544. Current Snapshot: MvccSnapshot[applied={T|T < 7332894275109494784}]
I20260924 13:43:11.388263 30412 mvcc.cc:205] Tried to move back new op lower bound from 7332894275109494784 to 7332894275039084544. Current Snapshot: MvccSnapshot[applied={T|T < 7332894275109494784}]
I20260924 13:43:11.389709 29916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:43:11.389801 29916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:43:11.390575 29916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c21a605549b4f3692cf3e75dcbc61ef: Sending DeleteTablet for 3 replicas of tablet f6b86340f98c459fa7396975b861553c
I20260924 13:43:11.390887 30314 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b86340f98c459fa7396975b861553c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:43:11 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:33892
I20260924 13:43:11.390931 30179 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b86340f98c459fa7396975b861553c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:43:11 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:58324
I20260924 13:43:11.391108 30044 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b86340f98c459fa7396975b861553c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:43:11 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:41806
I20260924 13:43:11.391161 30421 tablet_replica.cc:330] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73: stopping tablet replica
I20260924 13:43:11.391222 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 29980
I20260924 13:43:11.391412 30420 tablet_replica.cc:330] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc: stopping tablet replica
I20260924 13:43:11.391731 30421 raft_consensus.cc:2229] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:43:11.391865 30421 raft_consensus.cc:2258] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:43:11.392549 30421 ts_tablet_manager.cc:1905] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:43:11.393045 30420 raft_consensus.cc:2229] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:43:11.393211 30420 raft_consensus.cc:2258] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:43:11.393805 30420 ts_tablet_manager.cc:1905] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:43:11.394913 30421 ts_tablet_manager.cc:1918] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:43:11.395035 30421 log.cc:1196] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/f6b86340f98c459fa7396975b861553c
I20260924 13:43:11.395473 30421 ts_tablet_manager.cc:1939] T f6b86340f98c459fa7396975b861553c P b7db83a0716148eaa97a6aaf601c4e73: Deleting consensus metadata
I20260924 13:43:11.395506 30420 ts_tablet_manager.cc:1918] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:43:11.395643 30420 log.cc:1196] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/f6b86340f98c459fa7396975b861553c
I20260924 13:43:11.395922 29902 catalog_manager.cc:4841] TS b7db83a0716148eaa97a6aaf601c4e73 (127.19.53.194:41941): tablet f6b86340f98c459fa7396975b861553c (table test-table [id=c79f3c66985040b8acae51504059534f]) successfully deleted
I20260924 13:43:11.396086 30420 ts_tablet_manager.cc:1939] T f6b86340f98c459fa7396975b861553c P 3c99ca064bfb457caabbffb7245671bc: Deleting consensus metadata
I20260924 13:43:11.396476 29903 catalog_manager.cc:4841] TS 3c99ca064bfb457caabbffb7245671bc (127.19.53.195:42067): tablet f6b86340f98c459fa7396975b861553c (table test-table [id=c79f3c66985040b8acae51504059534f]) successfully deleted
W20260924 13:43:11.398368 29903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:35263 (error 108)
I20260924 13:43:11.398582 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 30115
I20260924 13:43:11.399276 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:11.398749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:35263 (local address 127.0.0.1:41816)
0924 13:43:11.399156 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:11.399240 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:35263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380}
W20260924 13:43:11.399410 29903 catalog_manager.cc:4584] TS 908c3c70c92e4e1bbd5a0d1bbbc2754b (127.19.53.193:35263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6b86340f98c459fa7396975b861553c: Network error: Client connection negotiation failed: client connection to 127.19.53.193:35263: connect: Connection refused (error 111)
I20260924 13:43:11.403707 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 30250
I20260924 13:43:11.408188 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 29885
2026-09-24T13:43:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1929 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:43:11 dist-test-slave-tfm9 krb5kdc[30426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:43:11 dist-test-slave-tfm9 krb5kdc[30426](info): set up 2 sockets
Sep 24 13:43:11 dist-test-slave-tfm9 krb5kdc[30426](info): commencing operation
krb5kdc: starting...
W20260924 13:43:12.436760 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:43:12 dist-test-slave-tfm9 krb5kdc[30426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:43:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:43:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/rangeradmin_127.19.53.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/rangeradmin_127.19.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/rangerlookup_127.19.53.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/rangerlookup_127.19.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/HTTP_127.19.53.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/HTTP_127.19.53.212@KRBTEST.COM.keytab.
I20260924 13:43:12.592403 19671 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrxQH3m/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/postgres -l logfile start
2026-09-24 13:43:15.049 UTC [30457] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 13:43:15.049 UTC [30457] LOG: listening on IPv4 address "127.19.53.212", port 38613
2026-09-24 13:43:15.052 UTC [30457] LOG: listening on Unix socket "/tmp/.s.PGSQL.38613"
2026-09-24 13:43:15.073 UTC [30460] LOG: database system was shut down at 2026-09-24 13:43:13 UTC
2026-09-24 13:43:15.077 UTC [30457] LOG: database system is ready to accept connections
I20260924 13:43:16.032342 19671 mini_postgres.cc:96] Postgres bound to 38613
2026-09-24 13:43:16.038 UTC [30468] FATAL: database "slave" does not exist
127.19.53.212:38613 - accepting connections
I20260924 13:43:16.039129 19671 mini_ranger.cc:161] Starting Ranger...
I20260924 13:43:16.053508 19671 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 13:43:16.499287 19671 mini_ranger.cc:87] Created ranger Postgres database
I20260924 13:43:16.499372 19671 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-admin
2026-09-24 13:43:16,765 [I] DB FLAVOR :POSTGRES
2026-09-24 13:43:16,765 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 13:43:16,765 [I] Checking connection..
2026-09-24 13:43:16,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 13:43:17,014 [I] Checking connection passed.
2026-09-24 13:43:17,015 [I] --------- Verifying version history table ---------
2026-09-24 13:43:17,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:43:17,283 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 13:43:17,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:43:17,514 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 13:43:17,514 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 13:43:17,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 13:43:17.758 UTC [30532] WARNING: there is no transaction in progress
2026-09-24 13:43:17,764 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 13:43:17,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:43:18,019 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 13:43:18,020 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 13:43:18,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 13:43:18,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 13:43:18,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tfm9.c.gcp-upstream.internal','N') ;"
2026-09-24 13:43:18,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 13:43:18,961 [I] Table x_portal_user does not exist in database ranger
2026-09-24 13:43:18,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 13:43:19,195 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 13:43:19,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 13:43:19,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 13:43:19,677 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 13:43:19,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 13:43:20.584 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.601 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.617 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.631 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.646 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.662 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.714 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.721 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.735 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.745 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.754 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.767 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.776 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.783 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.789 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:20.796 UTC [30641] WARNING: there is no transaction in progress
2026-09-24 13:43:21,400 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 13:43:21,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 13:43:21,635 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 13:43:21,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 13:43:21,874 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 13:43:21,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 13:43:22,114 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 13:43:22,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 13:43:22,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 13:43:22,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tfm9.c.gcp-upstream.internal';"
2026-09-24 13:43:22,781 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 13:43:22,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 13:43:23,039 [I] DB_PATCHES have already been applied
I20260924 13:43:23.044723 19671 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrxQH3m/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 13:43:23.044785 19671 mini_ranger.cc:193] Using host: 127.19.53.212
I20260924 13:43:23.047935 19671 mini_ranger.cc:227] Ranger admin URL: http://127.19.53.212:51269
Sep 24, 2026 1:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 1:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 1:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 1:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 1:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 13:43:24.260268 19671 mini_ranger.cc:160] Time spent starting Ranger: real 8.221s user 0.000s sys 0.009s
Sep 24, 2026 1:43:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 1:43:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 1:43:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 1:43:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 1:43:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 13:43:45.536814 19671 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/rangerkms_127.19.53.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/rangerkms_127.19.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.53.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.53.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.53.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/HTTP_127.19.53.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.53.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/HTTP_127.19.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 13:43:45 dist-test-slave-tfm9 krb5kdc[30426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257425, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 13:43:45.625483 19671 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 13:43:45.634810 19671 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 13:43:46.472602 19671 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 13:43:46.472680 19671 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-kms
I20260924 13:43:46.472697 19671 mini_ranger_kms.cc:227] Using postgres at 127.19.53.212:38613
2026-09-24 13:43:46,655 [I] DB FLAVOR :POSTGRES
2026-09-24 13:43:46,655 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 13:43:46,655 [I] Checking connection
2026-09-24 13:43:46,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 13:43:46,880 [I] connection success
2026-09-24 13:43:46,880 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 13:43:46,880 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 13:43:46,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 13:43:47,137 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 13:43:47,137 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 13:43:47,137 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 13:43:47,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:38613/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 13:43:47,398 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 13:43:47.875135 19671 mini_ranger_kms.cc:310] Created kms service
I20260924 13:43:48.077215 19671 mini_ranger_kms.cc:326] Created kudu user
I20260924 13:43:48.120496 19671 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 13:43:48 dist-test-slave-tfm9 krb5kdc[30426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257425, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:43:48.525274 19671 mini_ranger_kms.cc:384] Added ranger policy
I20260924 13:43:48.525403 19671 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 13:43:48.525427 19671 mini_ranger_kms.cc:240] Using host: 127.19.53.212
I20260924 13:43:48.528873 19671 mini_ranger_kms.cc:276] Ranger KMS PID: 30824
I20260924 13:43:48.529003 19671 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.53.212:59099
Sep 24, 2026 1:43:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:43:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:43:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 1:43:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 1:43:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 1:43:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 13:43:50.792305 19671 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.167s user 0.000s sys 0.166s
I20260924 13:43:50.792418 19671 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 13:43:50.792531 19671 mini_ranger_kms.cc:401] 127.19.53.212:59099/kms/v1/keys
Sep 24 13:43:51 dist-test-slave-tfm9 krb5kdc[30426](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790257431, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.53.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 13:43:51 dist-test-slave-tfm9 krb5kdc[30426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257431, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.53.212@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
Sep 24 13:43:51 dist-test-slave-tfm9 krb5kdc[30426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257431, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.53.212@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
Sep 24 13:43:53 dist-test-slave-tfm9 krb5kdc[30426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:43:53.944502 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33281
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:33281
--ranger_config_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.53.212:59099/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:43:54.062659 30875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:54.062942 30875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:54.062973 30875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:54.064566 30875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:43:54.064620 30875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:54.064639 30875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:54.064657 30875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:43:54.064673 30875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:43:54.066607 30875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.53.212:59099/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:33281
--ranger_config_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:54.066847 30875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:54.067090 30875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:54.071945 30880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:54.072052 30881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:54.071931 30883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:54.072504 30875 server_base.cc:1034] running on GCE node
I20260924 13:43:54.072736 30875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:54.073028 30875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:54.074183 30875 hybrid_clock.cc:648] HybridClock initialized: now 1790257434074156 us; error 40 us; skew 500 ppm
Sep 24 13:43:54 dist-test-slave-tfm9 krb5kdc[30426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257434, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:43:54.091882 30875 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:43:54.092433 30875 webserver.cc:460] Webserver started at http://127.19.53.254:33337/ using document root <none> and password file <none>
I20260924 13:43:54.092679 30875 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:54.092720 30875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:54.092854 30875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:43:54 dist-test-slave-tfm9 krb5kdc[30426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257434, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:43:54.150151 30875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "96346e1540144e7bbc6246cfd6a79ab9"
format_stamp: "Formatted at 2026-09-24 13:43:54 on dist-test-slave-tfm9"
server_key: "ba94851caad0fb8f81173e4df09df64e"
server_key_iv: "54d6a5e0a58f44dc098f28123cb3e462"
server_key_version: "kuduclusterkey@0"
I20260924 13:43:54.150595 30875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "96346e1540144e7bbc6246cfd6a79ab9"
format_stamp: "Formatted at 2026-09-24 13:43:54 on dist-test-slave-tfm9"
server_key: "ba94851caad0fb8f81173e4df09df64e"
server_key_iv: "54d6a5e0a58f44dc098f28123cb3e462"
server_key_version: "kuduclusterkey@0"
I20260924 13:43:54.152124 30875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:43:54.166534 30890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:54.166716 30875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:43:54.166821 30875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "96346e1540144e7bbc6246cfd6a79ab9"
format_stamp: "Formatted at 2026-09-24 13:43:54 on dist-test-slave-tfm9"
server_key: "ba94851caad0fb8f81173e4df09df64e"
server_key_iv: "54d6a5e0a58f44dc098f28123cb3e462"
server_key_version: "kuduclusterkey@0"
I20260924 13:43:54.166929 30875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:54.178943 30875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:54.181146 30875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:54.181289 30875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:54.186368 30875 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:33281
I20260924 13:43:54.186389 30943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:33281 every 8 connection(s)
I20260924 13:43:54.186810 30875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:43:54.187484 30944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:43:54.189752 30944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9: Bootstrap starting.
I20260924 13:43:54.190344 30944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9: Neither blocks nor log segments found. Creating new log.
I20260924 13:43:54.190631 30944 log.cc:824] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9: Log is configured to *not* fsync() on all Append() calls
I20260924 13:43:54.191174 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 30875
I20260924 13:43:54.191287 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:43:54.191413 30944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9: No bootstrap required, opened a new log
I20260924 13:43:54.191478 19671 external_mini_cluster.cc:1428] Setting key 90beaf3680fad1a5ab3d1467dab7dc64
I20260924 13:43:54.195174 30944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96346e1540144e7bbc6246cfd6a79ab9" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33281 } }
I20260924 13:43:54.195365 30944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:43:54.195405 30944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96346e1540144e7bbc6246cfd6a79ab9, State: Initialized, Role: FOLLOWER
I20260924 13:43:54.195555 30944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96346e1540144e7bbc6246cfd6a79ab9" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33281 } }
I20260924 13:43:54.195636 30944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:43:54.195663 30944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:43:54.195706 30944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:43:54.196421 30944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96346e1540144e7bbc6246cfd6a79ab9" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33281 } }
I20260924 13:43:54.196540 30944 leader_election.cc:302] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96346e1540144e7bbc6246cfd6a79ab9; no voters:
I20260924 13:43:54.196695 30944 leader_election.cc:288] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:43:54.196805 30949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:43:54.196931 30944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:43:54.196965 30949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 96346e1540144e7bbc6246cfd6a79ab9, State: Running, Role: LEADER
I20260924 13:43:54.197170 30949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96346e1540144e7bbc6246cfd6a79ab9" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33281 } }
I20260924 13:43:54.197505 30949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96346e1540144e7bbc6246cfd6a79ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96346e1540144e7bbc6246cfd6a79ab9" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33281 } } }
I20260924 13:43:54.197655 30949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [sys.catalog]: This master's current role is: LEADER
I20260924 13:43:54.197837 30949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96346e1540144e7bbc6246cfd6a79ab9. Latest consensus state: current_term: 1 leader_uuid: "96346e1540144e7bbc6246cfd6a79ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96346e1540144e7bbc6246cfd6a79ab9" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33281 } } }
I20260924 13:43:54.197963 30949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9 [sys.catalog]: This master's current role is: LEADER
I20260924 13:43:54.198150 30958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:43:54.198370 30944 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 13:43:54.198695 30958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 13:43:54 dist-test-slave-tfm9 krb5kdc[30426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:43:54.209540 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:54.191982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.19.53.254:33281)
0924 13:43:54.192162 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:54.192172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:54.192204 (+ 32us) server_negotiation.cc:407] Connection header received
0924 13:43:54.195148 (+ 2944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:54.195157 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:54.195213 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:54.195363 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:54.196181 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:54.197512 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:54.198646 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:54.201984 (+ 3338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:54.202194 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:54.205335 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:54.205340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:54.205340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:43:54.205352 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:54.207416 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:54.207842 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:54.207846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:54.207846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:54.207953 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:54.208158 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:54.208158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:54.208159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:54.208470 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:54.208633 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:54.209162 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 13:43:54.209290 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260924 13:43:54.211895 30958 catalog_manager.cc:1347] Generated new cluster ID: 8b2881c21bb544b4a05d730e6237a2ee
I20260924 13:43:54.211951 30958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:43:54.221426 30958 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:43:54.222259 30958 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:43:54.228226 30958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9: Generated new TSK 0
I20260924 13:43:54.228468 30958 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:43:54.972867 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.53.212:59099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33281
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:43:55.091823 30989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:55.092073 30989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:55.092111 30989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:55.093264 30989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:55.093326 30989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:55.093371 30989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:43:55.094944 30989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.53.212:59099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:55.095212 30989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:55.095445 30989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:55.101433 30997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:55.101465 30995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:55.101420 30994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:55.101863 30989 server_base.cc:1034] running on GCE node
I20260924 13:43:55.102094 30989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:55.102339 30989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:55.103513 30989 hybrid_clock.cc:648] HybridClock initialized: now 1790257435103487 us; error 32 us; skew 500 ppm
Sep 24 13:43:55 dist-test-slave-tfm9 krb5kdc[30426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257435, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:43:55.106386 30989 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:43:55.106869 30989 webserver.cc:460] Webserver started at http://127.19.53.193:37037/ using document root <none> and password file <none>
I20260924 13:43:55.107123 30989 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:55.107188 30989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:55.107333 30989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:43:55 dist-test-slave-tfm9 krb5kdc[30426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257435, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:43:55.121170 30989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "b1d91125f86c4cebbf0a6ad3356a42a3"
format_stamp: "Formatted at 2026-09-24 13:43:55 on dist-test-slave-tfm9"
server_key: "7ba513b56ac2a3d1aae70598b641f515"
server_key_iv: "a2beef17d8a34ddb7aade2bc5d9d1ccf"
server_key_version: "kuduclusterkey@0"
I20260924 13:43:55.121557 30989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "b1d91125f86c4cebbf0a6ad3356a42a3"
format_stamp: "Formatted at 2026-09-24 13:43:55 on dist-test-slave-tfm9"
server_key: "7ba513b56ac2a3d1aae70598b641f515"
server_key_iv: "a2beef17d8a34ddb7aade2bc5d9d1ccf"
server_key_version: "kuduclusterkey@0"
I20260924 13:43:55.123245 30989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:43:55.142056 31006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:55.142407 30989 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260924 13:43:55.142530 30989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "b1d91125f86c4cebbf0a6ad3356a42a3"
format_stamp: "Formatted at 2026-09-24 13:43:55 on dist-test-slave-tfm9"
server_key: "7ba513b56ac2a3d1aae70598b641f515"
server_key_iv: "a2beef17d8a34ddb7aade2bc5d9d1ccf"
server_key_version: "kuduclusterkey@0"
I20260924 13:43:55.142648 30989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:55.154752 30989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:55.157272 30989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:55.157423 30989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:55.157660 30989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:43:55.158077 30989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:43:55.158129 30989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:55.158160 30989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:43:55.158213 30989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:55.166352 30989 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:34411
I20260924 13:43:55.166445 31120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:34411 every 8 connection(s)
I20260924 13:43:55.166883 30989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:43:55.169957 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 30989
I20260924 13:43:55.170073 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:43:55.170292 19671 external_mini_cluster.cc:1428] Setting key 518f399f40e889fb80cd2fb29c6bdf3f
Sep 24 13:43:55 dist-test-slave-tfm9 krb5kdc[30426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257435, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
I20260924 13:43:55.184124 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:55.167718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:45855 (local address 127.19.53.254:33281)
0924 13:43:55.174046 (+ 6328us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:55.174048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:55.174066 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:43:55.174092 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:55.174092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:55.174106 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:55.174945 (+ 839us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:55.174986 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.175804 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.177308 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.177624 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.179379 (+ 1755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:55.180231 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:55.180232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:55.180233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:43:55.180239 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:55.182273 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:55.182824 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:55.182825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:55.182826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:55.183007 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:55.183331 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:55.183332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:55.183333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:55.183479 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:55.183559 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:55.183905 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 13:43:55.183994 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6294,"thread_start_us":79,"threads_started":1}
I20260924 13:43:55.184324 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:55.167677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33281 (local address 127.19.53.193:45855)
0924 13:43:55.167956 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:55.167999 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:55.168641 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:55.174313 (+ 5672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:55.174318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:55.174566 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:55.174849 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:55.174856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.175915 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.175916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:55.177208 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:55.177211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.177711 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.177712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:55.178030 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:55.178674 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:55.178685 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:55.180128 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:43:55.182362 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:43:55.182370 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:43:55.182372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:43:55.182729 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:43:55.183090 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:43:55.183091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:43:55.183091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:43:55.183248 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:43:55.183563 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:43:55.183569 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:43:55.183746 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 13:43:55.183868 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 13:43:55.184037 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":81,"threads_started":1}
I20260924 13:43:55.184892 31121 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33281
I20260924 13:43:55.185004 31121 heartbeater.cc:464] Registering TS with master...
I20260924 13:43:55.185190 31121 heartbeater.cc:511] Master 127.19.53.254:33281 requested a full tablet report, sending...
I20260924 13:43:55.185714 30907 ts_manager.cc:194] Registered new tserver with Master: b1d91125f86c4cebbf0a6ad3356a42a3 (127.19.53.193:34411)
I20260924 13:43:55.186507 30907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:45855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:43:55.198741 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.53.212:59099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33281
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:43:55 dist-test-slave-tfm9 krb5kdc[30426](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790257435, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 13:43:55.315339 31127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:55.315558 31127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:55.315594 31127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:55.316754 31127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:55.316860 31127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:55.316919 31127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:43:55.318506 31127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.53.212:59099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:55.318751 31127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:55.319010 31127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:55.323894 31137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:55.324334 31135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:55.324426 31134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:55.325588 31127 server_base.cc:1034] running on GCE node
I20260924 13:43:55.325867 31127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:55.326156 31127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:55.327322 31127 hybrid_clock.cc:648] HybridClock initialized: now 1790257435327301 us; error 39 us; skew 500 ppm
Sep 24 13:43:55 dist-test-slave-tfm9 krb5kdc[30426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257435, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:43:55.330617 31127 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:43:55.331115 31127 webserver.cc:460] Webserver started at http://127.19.53.194:42975/ using document root <none> and password file <none>
I20260924 13:43:55.331341 31127 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:55.331410 31127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:55.331552 31127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:43:55 dist-test-slave-tfm9 krb5kdc[30426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257435, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:43:55.345177 31127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "0241ffce64544b17882572a967b954d1"
format_stamp: "Formatted at 2026-09-24 13:43:55 on dist-test-slave-tfm9"
server_key: "3d01d44126c5588404f3b189fd912a00"
server_key_iv: "b5d4adfc62a58ac791a73cc28eabeb36"
server_key_version: "kuduclusterkey@0"
I20260924 13:43:55.345593 31127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "0241ffce64544b17882572a967b954d1"
format_stamp: "Formatted at 2026-09-24 13:43:55 on dist-test-slave-tfm9"
server_key: "3d01d44126c5588404f3b189fd912a00"
server_key_iv: "b5d4adfc62a58ac791a73cc28eabeb36"
server_key_version: "kuduclusterkey@0"
I20260924 13:43:55.350260 31127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 13:43:55.361896 31144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:55.362183 31127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:43:55.362282 31127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "0241ffce64544b17882572a967b954d1"
format_stamp: "Formatted at 2026-09-24 13:43:55 on dist-test-slave-tfm9"
server_key: "3d01d44126c5588404f3b189fd912a00"
server_key_iv: "b5d4adfc62a58ac791a73cc28eabeb36"
server_key_version: "kuduclusterkey@0"
I20260924 13:43:55.362378 31127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:55.382548 31127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:55.385217 31127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:55.385403 31127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:55.385699 31127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:43:55.386106 31127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:43:55.386173 31127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:55.386246 31127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:43:55.386283 31127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:55.393543 31127 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:34327
I20260924 13:43:55.394001 31127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:43:55.395985 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 31127
I20260924 13:43:55.396088 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:43:55.396317 19671 external_mini_cluster.cc:1428] Setting key 172bfe6b0cef72ae2ed99ba3d7bb002a
I20260924 13:43:55.403687 31259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:34327 every 8 connection(s)
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Sep 24 13:43:55 dist-test-slave-tfm9 krb5kdc[30426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257435, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:43:55.424731 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:55.404734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:59287 (local address 127.19.53.254:33281)
0924 13:43:55.404786 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:55.404788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:55.406360 (+ 1572us) server_negotiation.cc:407] Connection header received
0924 13:43:55.407101 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:55.407102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:55.407124 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:55.407162 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:55.408063 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.408873 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.410543 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.410859 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.410908 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:55.420815 (+ 9907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:55.420817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:55.420817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:43:55.420826 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:55.422876 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:55.423517 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:55.423519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:55.423519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:55.423586 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:55.423910 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:55.423911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:55.423912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:55.424077 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:55.424138 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:55.424524 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 13:43:55.424622 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:43:55.425546 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:55.404415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33281 (local address 127.19.53.194:59287)
0924 13:43:55.406219 (+ 1804us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:55.406268 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:55.406961 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:55.407362 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:55.407366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:55.407645 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:55.407939 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:55.407945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.409004 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.409005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:55.410421 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:55.410424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.411009 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.411010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:55.411155 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:55.417083 (+ 5928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:55.417095 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:55.420666 (+ 3571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:43:55.423067 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:43:55.423075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:43:55.423078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:43:55.423396 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:43:55.423666 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:43:55.423667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:43:55.423667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:43:55.423823 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:43:55.424175 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:43:55.424182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:43:55.424354 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 13:43:55.425113 (+ 759us) client_negotiation.cc:240] Negotiation successful
0924 13:43:55.425297 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1756,"thread_start_us":114,"threads_started":1}
I20260924 13:43:55.426270 31260 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33281
I20260924 13:43:55.426374 31260 heartbeater.cc:464] Registering TS with master...
I20260924 13:43:55.426573 31260 heartbeater.cc:511] Master 127.19.53.254:33281 requested a full tablet report, sending...
I20260924 13:43:55.427017 30907 ts_manager.cc:194] Registered new tserver with Master: 0241ffce64544b17882572a967b954d1 (127.19.53.194:34327)
I20260924 13:43:55.427505 30907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:59287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:43:55.436960 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.53.212:59099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33281
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:43:55.561748 31265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:43:55.562068 31265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:43:55.562110 31265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:43:55.564373 31265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:43:55.564443 31265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:43:55.564488 31265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:43:55.566896 31265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.53.212:59099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:43:55.567186 31265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:43:55.567492 31265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:43:55.574040 31271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:55.574179 31273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:43:55.574055 31270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:43:55.575652 31265 server_base.cc:1034] running on GCE node
I20260924 13:43:55.575913 31265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:43:55.576205 31265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:43:55.592370 31265 hybrid_clock.cc:648] HybridClock initialized: now 1790257435592306 us; error 64 us; skew 500 ppm
Sep 24 13:43:55 dist-test-slave-tfm9 krb5kdc[30426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257435, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:43:55.595278 31265 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:43:55.595836 31265 webserver.cc:460] Webserver started at http://127.19.53.195:40777/ using document root <none> and password file <none>
I20260924 13:43:55.596132 31265 fs_manager.cc:359] Metadata directory not provided
I20260924 13:43:55.596196 31265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:43:55.596364 31265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:43:55 dist-test-slave-tfm9 krb5kdc[30426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257435, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:43:55.613574 31265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "538b2020414543c1aae7aa28dbb2db13"
format_stamp: "Formatted at 2026-09-24 13:43:55 on dist-test-slave-tfm9"
server_key: "370926f86fa03a74879795072002b249"
server_key_iv: "187bf693875f8c82605433cd97652538"
server_key_version: "kuduclusterkey@0"
I20260924 13:43:55.614032 31265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "538b2020414543c1aae7aa28dbb2db13"
format_stamp: "Formatted at 2026-09-24 13:43:55 on dist-test-slave-tfm9"
server_key: "370926f86fa03a74879795072002b249"
server_key_iv: "187bf693875f8c82605433cd97652538"
server_key_version: "kuduclusterkey@0"
I20260924 13:43:55.629253 31265 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.002s sys 0.000s
I20260924 13:43:55.643113 31280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:55.654037 31265 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.000s sys 0.001s
I20260924 13:43:55.654184 31265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "538b2020414543c1aae7aa28dbb2db13"
format_stamp: "Formatted at 2026-09-24 13:43:55 on dist-test-slave-tfm9"
server_key: "370926f86fa03a74879795072002b249"
server_key_iv: "187bf693875f8c82605433cd97652538"
server_key_version: "kuduclusterkey@0"
I20260924 13:43:55.654297 31265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:43:55.680593 31265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:43:55.683245 31265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:43:55.683413 31265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:43:55.683692 31265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:43:55.684063 31265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:43:55.684114 31265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:55.684154 31265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:43:55.684185 31265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:43:55.692945 31265 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:43527
I20260924 13:43:55.693385 31265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:43:55.698616 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 31265
I20260924 13:43:55.698725 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:43:55.698961 19671 external_mini_cluster.cc:1428] Setting key 1d230cd2458a105eadbdbf2d0a289863
I20260924 13:43:55.700621 31393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:43527 every 8 connection(s)
Sep 24 13:43:55 dist-test-slave-tfm9 krb5kdc[30426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257435, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:43:55.713727 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:55.701941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:35765 (local address 127.19.53.254:33281)
0924 13:43:55.701983 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:55.701985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:55.702519 (+ 534us) server_negotiation.cc:407] Connection header received
0924 13:43:55.703237 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:55.703238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:55.703251 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:55.703281 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:55.704568 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.705335 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.706878 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.707180 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.707225 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:55.709873 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:55.709875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:55.709875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:43:55.709882 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:55.712200 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:55.712704 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:55.712705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:55.712706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:55.712761 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:55.713015 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:55.713015 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:55.713016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:55.713140 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:55.713180 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:55.713547 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 13:43:55.713630 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:43:55.714286 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:55.701644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33281 (local address 127.19.53.195:35765)
0924 13:43:55.702386 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:55.702429 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:55.703108 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:55.703466 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:55.703470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:55.703725 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:43:55.704434 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:55.704442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.705463 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.705466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:55.706781 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:55.706784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.707314 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.707315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:55.707458 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:55.708094 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:43:55.708104 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:43:55.709731 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:43:55.712296 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:43:55.712302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:43:55.712305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:43:55.712615 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:43:55.712828 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:43:55.712829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:43:55.712829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:43:55.712945 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:43:55.713223 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:43:55.713228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:43:55.713386 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 13:43:55.713871 (+ 485us) client_negotiation.cc:240] Negotiation successful
0924 13:43:55.714053 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":90,"threads_started":1}
I20260924 13:43:55.714708 31394 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33281
I20260924 13:43:55.714804 31394 heartbeater.cc:464] Registering TS with master...
I20260924 13:43:55.714996 31394 heartbeater.cc:511] Master 127.19.53.254:33281 requested a full tablet report, sending...
I20260924 13:43:55.715325 30907 ts_manager.cc:194] Registered new tserver with Master: 538b2020414543c1aae7aa28dbb2db13 (127.19.53.195:43527)
I20260924 13:43:55.715816 30907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:35765
I20260924 13:43:55.721459 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:43:55.738094 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:55.730182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.19.53.254:33281)
0924 13:43:55.730225 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:55.730227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:55.730579 (+ 352us) server_negotiation.cc:407] Connection header received
0924 13:43:55.730680 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:55.730681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:55.730692 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:55.730721 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:43:55.731502 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.732258 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.733332 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.733629 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.733667 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:55.735240 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:43:55.735241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:43:55.735242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:43:55.735248 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:43:55.736874 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:55.737262 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:55.737263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:55.737263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:55.737312 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:43:55.737516 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:43:55.737517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:43:55.737517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:43:55.737633 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:43:55.737703 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:55.737899 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 13:43:55.737999 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:43:55.740005 30907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:43:55.742162 30907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:43:55.757783 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:55.748280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:34327 (local address 127.0.0.1:60298)
0924 13:43:55.750393 (+ 2113us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:55.750401 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:55.750437 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:55.750859 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:55.750862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:55.750864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:43:55.750935 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:55.750936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.752247 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.752248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:55.753564 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:55.753566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.757459 (+ 3893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.757459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:55.757570 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:55.757586 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 13:43:55.757648 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 13:43:55.757697 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2101,"thread_start_us":47,"threads_started":1}
I20260924 13:43:55.758359 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:55.748409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.19.53.194:34327)
0924 13:43:55.750105 (+ 1696us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:55.750111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:55.750491 (+ 380us) server_negotiation.cc:407] Connection header received
0924 13:43:55.750537 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:55.750540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:55.750558 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:55.750629 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:43:55.751016 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.752152 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.753673 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.754458 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.754527 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:55.754600 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:55.758230 (+ 3630us) server_negotiation.cc:299] Negotiation successful
0924 13:43:55.758278 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1652,"thread_start_us":85,"threads_started":1}
I20260924 13:43:55.759528 31193 tablet_service.cc:1467] Processing CreateTablet for tablet 1d522e6bf7004c34a06c5c567f6c6deb (DEFAULT_TABLE table=test-table [id=92ee674bf36440c4902be32f685d5f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:43:55.759896 31193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d522e6bf7004c34a06c5c567f6c6deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:43:55.762089 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:55.748431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:34411 (local address 127.0.0.1:59644)
0924 13:43:55.750740 (+ 2309us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:55.750750 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:55.750792 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:55.758280 (+ 7488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:55.758281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:55.758283 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:43:55.758320 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:55.758321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.759609 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.759610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:55.760793 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:55.760796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.761827 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.761828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:55.761892 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:55.761896 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:43:55.761964 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 13:43:55.762014 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2294,"thread_start_us":60,"threads_started":1}
I20260924 13:43:55.762373 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:55.748884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.19.53.193:34411)
0924 13:43:55.758032 (+ 9148us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:55.758037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:55.758050 (+ 13us) server_negotiation.cc:407] Connection header received
0924 13:43:55.758086 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:55.758088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:55.758105 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:55.758199 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:43:55.758404 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.759501 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.760908 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.761668 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.761738 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:55.761793 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:55.762263 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 13:43:55.762306 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9108,"thread_start_us":66,"threads_started":1}
I20260924 13:43:55.762959 31054 tablet_service.cc:1467] Processing CreateTablet for tablet 1d522e6bf7004c34a06c5c567f6c6deb (DEFAULT_TABLE table=test-table [id=92ee674bf36440c4902be32f685d5f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:43:55.763317 31054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d522e6bf7004c34a06c5c567f6c6deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:43:55.765765 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:55.748082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:43527 (local address 127.0.0.1:39118)
0924 13:43:55.750173 (+ 2091us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:55.750217 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:55.750290 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:55.754925 (+ 4635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:55.754926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:55.754928 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:43:55.754972 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:55.754973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.756224 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.756225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:55.757353 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:55.757355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.765512 (+ 8157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.765513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:55.765588 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:55.765592 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:43:55.765639 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 13:43:55.765695 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2063,"thread_start_us":66,"threads_started":1}
I20260924 13:43:55.766304 31411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:55.748695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.19.53.195:43527)
0924 13:43:55.754699 (+ 6004us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:55.754704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:55.754714 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:43:55.754755 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:55.754757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:55.754774 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:55.754846 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:43:55.755056 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.756136 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.764538 (+ 8402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:55.765348 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:55.765415 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:55.765479 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:55.766128 (+ 649us) server_negotiation.cc:299] Negotiation successful
0924 13:43:55.766173 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5964,"thread_start_us":95,"threads_started":1}
I20260924 13:43:55.767023 31309 tablet_service.cc:1467] Processing CreateTablet for tablet 1d522e6bf7004c34a06c5c567f6c6deb (DEFAULT_TABLE table=test-table [id=92ee674bf36440c4902be32f685d5f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:43:55.767323 31309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d522e6bf7004c34a06c5c567f6c6deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:43:55.774109 31414 tablet_bootstrap.cc:492] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3: Bootstrap starting.
I20260924 13:43:55.774353 31413 tablet_bootstrap.cc:492] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1: Bootstrap starting.
I20260924 13:43:55.776129 31414 tablet_bootstrap.cc:654] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3: Neither blocks nor log segments found. Creating new log.
I20260924 13:43:55.776427 31414 log.cc:824] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3: Log is configured to *not* fsync() on all Append() calls
I20260924 13:43:55.776762 31415 tablet_bootstrap.cc:492] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13: Bootstrap starting.
I20260924 13:43:55.780103 31413 tablet_bootstrap.cc:654] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1: Neither blocks nor log segments found. Creating new log.
I20260924 13:43:55.780476 31413 log.cc:824] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1: Log is configured to *not* fsync() on all Append() calls
I20260924 13:43:55.782087 31415 tablet_bootstrap.cc:654] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13: Neither blocks nor log segments found. Creating new log.
I20260924 13:43:55.782454 31415 log.cc:824] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13: Log is configured to *not* fsync() on all Append() calls
I20260924 13:43:55.783382 31414 tablet_bootstrap.cc:492] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3: No bootstrap required, opened a new log
I20260924 13:43:55.783509 31414 ts_tablet_manager.cc:1397] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 13:43:55.786522 31414 raft_consensus.cc:348] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538b2020414543c1aae7aa28dbb2db13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43527 } } peers { permanent_uuid: "b1d91125f86c4cebbf0a6ad3356a42a3" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34411 } } peers { permanent_uuid: "0241ffce64544b17882572a967b954d1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 34327 } }
I20260924 13:43:55.786809 31414 raft_consensus.cc:374] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:43:55.786888 31414 raft_consensus.cc:729] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1d91125f86c4cebbf0a6ad3356a42a3, State: Initialized, Role: FOLLOWER
I20260924 13:43:55.787109 31414 consensus_queue.cc:260] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538b2020414543c1aae7aa28dbb2db13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43527 } } peers { permanent_uuid: "b1d91125f86c4cebbf0a6ad3356a42a3" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34411 } } peers { permanent_uuid: "0241ffce64544b17882572a967b954d1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 34327 } }
I20260924 13:43:55.787617 31414 ts_tablet_manager.cc:1428] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 13:43:55.787793 31121 heartbeater.cc:503] Master 127.19.53.254:33281 was elected leader, sending a full tablet report...
I20260924 13:43:55.789719 31413 tablet_bootstrap.cc:492] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1: No bootstrap required, opened a new log
I20260924 13:43:55.789793 31413 ts_tablet_manager.cc:1397] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260924 13:43:55.791769 31413 raft_consensus.cc:348] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538b2020414543c1aae7aa28dbb2db13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43527 } } peers { permanent_uuid: "b1d91125f86c4cebbf0a6ad3356a42a3" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34411 } } peers { permanent_uuid: "0241ffce64544b17882572a967b954d1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 34327 } }
I20260924 13:43:55.791921 31413 raft_consensus.cc:374] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:43:55.791949 31413 raft_consensus.cc:729] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0241ffce64544b17882572a967b954d1, State: Initialized, Role: FOLLOWER
I20260924 13:43:55.792399 31413 consensus_queue.cc:260] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538b2020414543c1aae7aa28dbb2db13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43527 } } peers { permanent_uuid: "b1d91125f86c4cebbf0a6ad3356a42a3" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34411 } } peers { permanent_uuid: "0241ffce64544b17882572a967b954d1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 34327 } }
I20260924 13:43:55.792737 31413 ts_tablet_manager.cc:1428] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:43:55.792886 31260 heartbeater.cc:503] Master 127.19.53.254:33281 was elected leader, sending a full tablet report...
I20260924 13:43:55.794126 31415 tablet_bootstrap.cc:492] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13: No bootstrap required, opened a new log
I20260924 13:43:55.794196 31415 ts_tablet_manager.cc:1397] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260924 13:43:55.796079 31415 raft_consensus.cc:348] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538b2020414543c1aae7aa28dbb2db13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43527 } } peers { permanent_uuid: "b1d91125f86c4cebbf0a6ad3356a42a3" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34411 } } peers { permanent_uuid: "0241ffce64544b17882572a967b954d1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 34327 } }
I20260924 13:43:55.796240 31415 raft_consensus.cc:374] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:43:55.796271 31415 raft_consensus.cc:729] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538b2020414543c1aae7aa28dbb2db13, State: Initialized, Role: FOLLOWER
I20260924 13:43:55.796371 31415 consensus_queue.cc:260] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538b2020414543c1aae7aa28dbb2db13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43527 } } peers { permanent_uuid: "b1d91125f86c4cebbf0a6ad3356a42a3" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34411 } } peers { permanent_uuid: "0241ffce64544b17882572a967b954d1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 34327 } }
I20260924 13:43:55.796609 31415 ts_tablet_manager.cc:1428] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:43:55.796815 31394 heartbeater.cc:503] Master 127.19.53.254:33281 was elected leader, sending a full tablet report...
Sep 24 13:43:55 dist-test-slave-tfm9 krb5kdc[30426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257435, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
W20260924 13:43:55.905285 31261 tablet.cc:2369] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:43:55.919412 31122 tablet.cc:2369] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:43:55.952342 31395 tablet.cc:2369] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:43:56.033643 31421 raft_consensus.cc:482] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:43:56.033811 31421 raft_consensus.cc:504] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538b2020414543c1aae7aa28dbb2db13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43527 } } peers { permanent_uuid: "b1d91125f86c4cebbf0a6ad3356a42a3" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34411 } } peers { permanent_uuid: "0241ffce64544b17882572a967b954d1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 34327 } }
I20260924 13:43:56.034276 31421 leader_election.cc:288] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1d91125f86c4cebbf0a6ad3356a42a3 (127.19.53.193:34411), 0241ffce64544b17882572a967b954d1 (127.19.53.194:34327)
I20260924 13:43:56.039114 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:56.034689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:34411 (local address 127.19.53.195:55047)
0924 13:43:56.034799 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:56.034811 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:56.034872 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:56.035549 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:56.035550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:56.035552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:43:56.035626 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:56.035628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:56.036651 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:56.036653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:56.037893 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:56.037896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:56.038826 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:56.038828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:56.038914 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:56.038918 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:43:56.038984 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 13:43:56.039028 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 13:43:56.039904 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:56.035327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:55047 (local address 127.19.53.193:34411)
0924 13:43:56.035375 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:56.035377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:56.035387 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:43:56.035416 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:56.035417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:56.035427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:56.035491 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:43:56.035721 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:56.036535 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:56.038035 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:56.038672 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:56.038752 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:56.038762 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:56.039788 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0924 13:43:56.039837 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:43:56.040220 31074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d522e6bf7004c34a06c5c567f6c6deb" candidate_uuid: "538b2020414543c1aae7aa28dbb2db13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1d91125f86c4cebbf0a6ad3356a42a3" is_pre_election: true
I20260924 13:43:56.040401 31074 raft_consensus.cc:2454] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 538b2020414543c1aae7aa28dbb2db13 in term 0.
I20260924 13:43:56.041441 31282 leader_election.cc:302] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 538b2020414543c1aae7aa28dbb2db13, b1d91125f86c4cebbf0a6ad3356a42a3; no voters:
I20260924 13:43:56.041647 31421 raft_consensus.cc:2781] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:43:56.041703 31421 raft_consensus.cc:482] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:43:56.041728 31421 raft_consensus.cc:3037] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:43:56.042800 31421 raft_consensus.cc:504] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538b2020414543c1aae7aa28dbb2db13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43527 } } peers { permanent_uuid: "b1d91125f86c4cebbf0a6ad3356a42a3" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34411 } } peers { permanent_uuid: "0241ffce64544b17882572a967b954d1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 34327 } }
I20260924 13:43:56.042977 31421 leader_election.cc:288] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [CANDIDATE]: Term 1 election: Requested vote from peers b1d91125f86c4cebbf0a6ad3356a42a3 (127.19.53.193:34411), 0241ffce64544b17882572a967b954d1 (127.19.53.194:34327)
I20260924 13:43:56.043254 31074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d522e6bf7004c34a06c5c567f6c6deb" candidate_uuid: "538b2020414543c1aae7aa28dbb2db13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1d91125f86c4cebbf0a6ad3356a42a3"
I20260924 13:43:56.043337 31074 raft_consensus.cc:3037] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:43:56.044486 31074 raft_consensus.cc:2454] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 538b2020414543c1aae7aa28dbb2db13 in term 1.
I20260924 13:43:56.044719 31282 leader_election.cc:302] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 538b2020414543c1aae7aa28dbb2db13, b1d91125f86c4cebbf0a6ad3356a42a3; no voters:
I20260924 13:43:56.044839 31421 raft_consensus.cc:2781] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:43:56.045012 31421 raft_consensus.cc:686] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [term 1 LEADER]: Becoming Leader. State: Replica: 538b2020414543c1aae7aa28dbb2db13, State: Running, Role: LEADER
I20260924 13:43:56.045102 31421 consensus_queue.cc:237] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538b2020414543c1aae7aa28dbb2db13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43527 } } peers { permanent_uuid: "b1d91125f86c4cebbf0a6ad3356a42a3" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34411 } } peers { permanent_uuid: "0241ffce64544b17882572a967b954d1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 34327 } }
I20260924 13:43:56.046002 30907 catalog_manager.cc:5495] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 538b2020414543c1aae7aa28dbb2db13 (127.19.53.195). New cstate: current_term: 1 leader_uuid: "538b2020414543c1aae7aa28dbb2db13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538b2020414543c1aae7aa28dbb2db13" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1d91125f86c4cebbf0a6ad3356a42a3" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0241ffce64544b17882572a967b954d1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 34327 } health_report { overall_health: UNKNOWN } } }
I20260924 13:43:56.048151 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:56.035058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:34327 (local address 127.19.53.195:33613)
0924 13:43:56.039194 (+ 4136us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:56.039207 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:43:56.039249 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:43:56.040270 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:43:56.040271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:43:56.040273 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:43:56.040310 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:56.040311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:56.045645 (+ 5334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:56.045646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:56.047023 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:43:56.047026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:43:56.047906 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:43:56.047907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:43:56.047979 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:56.047982 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:43:56.048034 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 13:43:56.048071 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4125,"thread_start_us":100,"threads_started":1}
I20260924 13:43:56.048458 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:56.035258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:33613 (local address 127.19.53.194:34327)
0924 13:43:56.035320 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:56.035323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:56.039313 (+ 3990us) server_negotiation.cc:407] Connection header received
0924 13:43:56.039344 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:56.039346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:56.039360 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:56.039409 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:43:56.040417 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:56.041105 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:56.047136 (+ 6031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:56.047729 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:56.047805 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:56.047814 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:56.048349 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 13:43:56.048391 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 13:43:56.049047 31206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d522e6bf7004c34a06c5c567f6c6deb" candidate_uuid: "538b2020414543c1aae7aa28dbb2db13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0241ffce64544b17882572a967b954d1" is_pre_election: true
I20260924 13:43:56.049199 31206 raft_consensus.cc:2454] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 538b2020414543c1aae7aa28dbb2db13 in term 0.
I20260924 13:43:56.049305 31205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d522e6bf7004c34a06c5c567f6c6deb" candidate_uuid: "538b2020414543c1aae7aa28dbb2db13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0241ffce64544b17882572a967b954d1"
I20260924 13:43:56.049355 31205 raft_consensus.cc:3037] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:43:56.050460 31205 raft_consensus.cc:2454] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 538b2020414543c1aae7aa28dbb2db13 in term 1.
I20260924 13:43:56.109771 31411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:56.101276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39126 (local address 127.19.53.195:43527)
0924 13:43:56.101331 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:43:56.101334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:43:56.106060 (+ 4726us) server_negotiation.cc:407] Connection header received
0924 13:43:56.106165 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:43:56.106167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:43:56.106191 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:43:56.106298 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:43:56.106689 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:56.107423 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:56.108570 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:43:56.108893 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:43:56.108958 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:43:56.109166 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:43:56.109308 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:43:56.109385 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 13:43:56.109644 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 13:43:56.109666 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 13:43:56.114902 31074 raft_consensus.cc:1261] T 1d522e6bf7004c34a06c5c567f6c6deb P b1d91125f86c4cebbf0a6ad3356a42a3 [term 1 FOLLOWER]: Refusing update from remote peer 538b2020414543c1aae7aa28dbb2db13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:43:56.115231 31205 raft_consensus.cc:1261] T 1d522e6bf7004c34a06c5c567f6c6deb P 0241ffce64544b17882572a967b954d1 [term 1 FOLLOWER]: Refusing update from remote peer 538b2020414543c1aae7aa28dbb2db13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:43:56.115712 31423 consensus_queue.cc:1035] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0241ffce64544b17882572a967b954d1" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:43:56.116192 31423 consensus_queue.cc:1035] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1d91125f86c4cebbf0a6ad3356a42a3" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 34411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 24 13:43:56 dist-test-slave-tfm9 krb5kdc[30426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257435, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:43:56.121464 31427 mvcc.cc:205] Tried to move back new op lower bound from 7332894458314481664 to 7332894458041094144. Current Snapshot: MvccSnapshot[applied={T|T < 7332894458314481664}]
I20260924 13:43:56.122381 31431 mvcc.cc:205] Tried to move back new op lower bound from 7332894458314481664 to 7332894458041094144. Current Snapshot: MvccSnapshot[applied={T|T < 7332894458314481664}]
I20260924 13:43:56.124518 31428 mvcc.cc:205] Tried to move back new op lower bound from 7332894458314481664 to 7332894458041094144. Current Snapshot: MvccSnapshot[applied={T|T < 7332894458314481664}]
I20260924 13:43:56.127529 30907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:43:56.127619 30907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:43:56.130293 30907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96346e1540144e7bbc6246cfd6a79ab9: Sending DeleteTablet for 3 replicas of tablet 1d522e6bf7004c34a06c5c567f6c6deb
I20260924 13:43:56.131315 31309 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d522e6bf7004c34a06c5c567f6c6deb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:43:56 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:39118
I20260924 13:43:56.131518 31193 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d522e6bf7004c34a06c5c567f6c6deb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:43:56 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:60298
I20260924 13:43:56.132090 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 30989
I20260924 13:43:56.132088 31054 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d522e6bf7004c34a06c5c567f6c6deb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:43:56 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:59644
W20260924 13:43:56.140457 30892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:34411 (error 108)
I20260924 13:43:56.141062 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 31127
I20260924 13:43:56.149801 31438 tablet_replica.cc:330] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13: stopping tablet replica
I20260924 13:43:56.154053 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:56.153885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:34411 (local address 127.0.0.1:59660)
0924 13:43:56.153973 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:56.154012 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:34411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 13:43:56.154325 30892 catalog_manager.cc:4584] TS b1d91125f86c4cebbf0a6ad3356a42a3 (127.19.53.193:34411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d522e6bf7004c34a06c5c567f6c6deb: Network error: Client connection negotiation failed: client connection to 127.19.53.193:34411: connect: Connection refused (error 111)
I20260924 13:43:56.154616 31438 raft_consensus.cc:2229] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:43:56.154870 31438 raft_consensus.cc:2258] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:43:56.155464 31438 ts_tablet_manager.cc:1905] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:43:56.157428 31438 ts_tablet_manager.cc:1918] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:43:56.157560 31438 log.cc:1196] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/1d522e6bf7004c34a06c5c567f6c6deb
I20260924 13:43:56.157892 31438 ts_tablet_manager.cc:1939] T 1d522e6bf7004c34a06c5c567f6c6deb P 538b2020414543c1aae7aa28dbb2db13: Deleting consensus metadata
I20260924 13:43:56.158542 30891 catalog_manager.cc:4841] TS 538b2020414543c1aae7aa28dbb2db13 (127.19.53.195:43527): tablet 1d522e6bf7004c34a06c5c567f6c6deb (table test-table [id=92ee674bf36440c4902be32f685d5f65]) successfully deleted
I20260924 13:43:56.158699 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 31265
I20260924 13:43:56.160053 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:43:56.159912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:34327 (local address 127.0.0.1:60300)
0924 13:43:56.159982 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 13:43:56.160018 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.194:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 13:43:56.168519 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 30875
I20260924 13:43:56.175348 19671 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 13:43:56.519693 19671 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 13:43:56.519800 19671 mini_ranger.cc:66] Stopping Ranger...
I20260924 13:43:56.863845 19671 mini_ranger.cc:68] Stopped Ranger
2026-09-24 13:43:56.864 UTC [30457] LOG: received smart shutdown request
2026-09-24 13:43:56.867 UTC [30457] LOG: background worker "logical replication launcher" (PID 30466) exited with exit code 1
2026-09-24 13:43:56.868 UTC [30461] LOG: shutting down
2026-09-24 13:43:57.023 UTC [30457] LOG: database system is shut down
2026-09-24T13:43:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45717 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:43:57 dist-test-slave-tfm9 krb5kdc[31450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:43:57 dist-test-slave-tfm9 krb5kdc[31450](info): set up 2 sockets
Sep 24 13:43:57 dist-test-slave-tfm9 krb5kdc[31450](info): commencing operation
krb5kdc: starting...
W20260924 13:43:58.179414 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:43:58 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:43:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/rangeradmin_127.19.53.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/rangeradmin_127.19.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/rangerlookup_127.19.53.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/rangerlookup_127.19.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/HTTP_127.19.53.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/HTTP_127.19.53.212@KRBTEST.COM.keytab.
I20260924 13:43:58.332602 19671 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrxQH3m/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/postgres -l logfile start
2026-09-24 13:44:00.740 UTC [31480] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 13:44:00.740 UTC [31480] LOG: listening on IPv4 address "127.19.53.212", port 42327
2026-09-24 13:44:00.743 UTC [31480] LOG: listening on Unix socket "/tmp/.s.PGSQL.42327"
2026-09-24 13:44:00.779 UTC [31483] LOG: database system was shut down at 2026-09-24 13:43:59 UTC
2026-09-24 13:44:00.782 UTC [31480] LOG: database system is ready to accept connections
I20260924 13:44:01.735388 19671 mini_postgres.cc:96] Postgres bound to 42327
2026-09-24 13:44:01.741 UTC [31491] FATAL: database "slave" does not exist
127.19.53.212:42327 - accepting connections
I20260924 13:44:01.741727 19671 mini_ranger.cc:161] Starting Ranger...
I20260924 13:44:01.756155 19671 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 13:44:02.217090 19671 mini_ranger.cc:87] Created ranger Postgres database
I20260924 13:44:02.217160 19671 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-admin
2026-09-24 13:44:02,512 [I] DB FLAVOR :POSTGRES
2026-09-24 13:44:02,513 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 13:44:02,513 [I] Checking connection..
2026-09-24 13:44:02,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 13:44:02,756 [I] Checking connection passed.
2026-09-24 13:44:02,757 [I] --------- Verifying version history table ---------
2026-09-24 13:44:02,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:44:02,998 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 13:44:02,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:44:03,256 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 13:44:03,256 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 13:44:03,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 13:44:03.515 UTC [31557] WARNING: there is no transaction in progress
2026-09-24 13:44:03,521 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 13:44:03,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:44:03,790 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 13:44:03,790 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 13:44:03,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 13:44:04,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 13:44:04,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tfm9.c.gcp-upstream.internal','N') ;"
2026-09-24 13:44:04,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 13:44:04,773 [I] Table x_portal_user does not exist in database ranger
2026-09-24 13:44:04,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 13:44:05,010 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 13:44:05,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 13:44:05,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 13:44:05,494 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 13:44:05,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 13:44:06.406 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.422 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.438 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.453 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.468 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.485 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.537 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.543 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.557 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.568 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.578 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.591 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.600 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.606 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.613 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:06.619 UTC [31667] WARNING: there is no transaction in progress
2026-09-24 13:44:07,244 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 13:44:07,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 13:44:07,464 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 13:44:07,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 13:44:07,742 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 13:44:07,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 13:44:07,979 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 13:44:07,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 13:44:08,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 13:44:08,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tfm9.c.gcp-upstream.internal';"
2026-09-24 13:44:08,687 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 13:44:08,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 13:44:08,918 [I] DB_PATCHES have already been applied
I20260924 13:44:08.924311 19671 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrxQH3m/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 13:44:08.924373 19671 mini_ranger.cc:193] Using host: 127.19.53.212
I20260924 13:44:08.927439 19671 mini_ranger.cc:227] Ranger admin URL: http://127.19.53.212:44283
Sep 24, 2026 1:44:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:44:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:44:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 1:44:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 1:44:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 1:44:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 1:44:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 13:44:10.022644 19671 mini_ranger.cc:160] Time spent starting Ranger: real 8.281s user 0.003s sys 0.005s
Sep 24, 2026 1:44:30 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 1:44:30 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 1:44:30 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 1:44:30 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 1:44:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 1:44:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 1:44:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 1:44:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 1:44:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 1:44:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 1:44:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 1:44:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 1:44:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 1:44:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 1:44:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 1:44:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 1:44:30 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 1:44:31 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 13:44:31.911832 19671 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/rangerkms_127.19.53.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/rangerkms_127.19.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.53.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.53.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.53.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/HTTP_127.19.53.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.53.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/HTTP_127.19.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 13:44:31 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257471, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 13:44:31.994100 19671 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 13:44:32.003337 19671 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 13:44:32.861816 19671 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 13:44:32.861912 19671 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-kms
I20260924 13:44:32.861963 19671 mini_ranger_kms.cc:227] Using postgres at 127.19.53.212:42327
2026-09-24 13:44:33,035 [I] DB FLAVOR :POSTGRES
2026-09-24 13:44:33,035 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 13:44:33,035 [I] Checking connection
2026-09-24 13:44:33,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 13:44:33,263 [I] connection success
2026-09-24 13:44:33,264 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 13:44:33,264 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 13:44:33,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 13:44:33,490 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 13:44:33,490 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 13:44:33,490 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 13:44:33,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.53.212:42327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 13:44:33,751 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 13:44:34.250737 19671 mini_ranger_kms.cc:310] Created kms service
I20260924 13:44:34.459689 19671 mini_ranger_kms.cc:326] Created kudu user
I20260924 13:44:34.504309 19671 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 13:44:34 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257471, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:44:34.892419 19671 mini_ranger_kms.cc:384] Added ranger policy
I20260924 13:44:34.892624 19671 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrxQH3m/build/release/bin/postgresql.jar:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrxQH3m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 13:44:34.892643 19671 mini_ranger_kms.cc:240] Using host: 127.19.53.212
I20260924 13:44:34.896111 19671 mini_ranger_kms.cc:276] Ranger KMS PID: 31850
I20260924 13:44:34.896190 19671 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.53.212:51933
Sep 24, 2026 1:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 1:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 1:44:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 1:44:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 13:44:37 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790257477, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.53.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:37.196362 19671 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.202s user 0.000s sys 0.157s
I20260924 13:44:37.196455 19671 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 13:44:37.196502 19671 mini_ranger_kms.cc:401] 127.19.53.212:51933/kms/v1/keys
Sep 24 13:44:37 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257477, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.53.212@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
Sep 24 13:44:37 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257477, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.53.212@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
Sep 24 13:44:40 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:44:40.101799 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33465
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:33465
--ranger_config_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.53.212:51933/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:44:40.195415 31902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:40.195606 31902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:40.195627 31902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:40.196794 31902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:44:40.196835 31902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:40.196849 31902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:40.196861 31902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:44:40.196873 31902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:44:40.198489 31902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.53.212:51933/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:33465
--ranger_config_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:40.198716 31902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:40.198958 31902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:40.204144 31907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:40.204276 31902 server_base.cc:1034] running on GCE node
W20260924 13:44:40.204137 31910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:40.204119 31908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:40.204671 31902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:40.205009 31902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:40.206174 31902 hybrid_clock.cc:648] HybridClock initialized: now 1790257480206152 us; error 44 us; skew 500 ppm
Sep 24 13:44:40 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257480, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:40.209268 31902 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:44:40.209787 31902 webserver.cc:460] Webserver started at http://127.19.53.254:41225/ using document root <none> and password file <none>
I20260924 13:44:40.210042 31902 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:40.210096 31902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:40.210229 31902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:44:40 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257480, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:44:40.255766 31902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "540ddda89e104574acc48bceedc7cf35"
format_stamp: "Formatted at 2026-09-24 13:44:40 on dist-test-slave-tfm9"
server_key: "a319ad58528b220f6091c10d0c1f50ea"
server_key_iv: "3c53b24bc07e701e10f4a179648a7723"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:40.256196 31902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "540ddda89e104574acc48bceedc7cf35"
format_stamp: "Formatted at 2026-09-24 13:44:40 on dist-test-slave-tfm9"
server_key: "a319ad58528b220f6091c10d0c1f50ea"
server_key_iv: "3c53b24bc07e701e10f4a179648a7723"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:40.258081 31902 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 13:44:40.269606 31917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:40.269814 31902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:40.269915 31902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "540ddda89e104574acc48bceedc7cf35"
format_stamp: "Formatted at 2026-09-24 13:44:40 on dist-test-slave-tfm9"
server_key: "a319ad58528b220f6091c10d0c1f50ea"
server_key_iv: "3c53b24bc07e701e10f4a179648a7723"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:40.270051 31902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:40.286060 31902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:40.288208 31902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:40.288399 31902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:40.293148 31902 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:33465
I20260924 13:44:40.293224 31970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:33465 every 8 connection(s)
I20260924 13:44:40.293563 31902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:44:40.294322 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:40.297348 31971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35: Bootstrap starting.
I20260924 13:44:40.297883 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 31902
I20260924 13:44:40.298029 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:44:40.298090 31971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:40.298228 19671 external_mini_cluster.cc:1428] Setting key 8933877278a108254abbeb2726357ac0
I20260924 13:44:40.298399 31971 log.cc:824] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:40.299180 31971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35: No bootstrap required, opened a new log
I20260924 13:44:40.301719 31971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540ddda89e104574acc48bceedc7cf35" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33465 } }
I20260924 13:44:40.301978 31971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:40.302019 31971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540ddda89e104574acc48bceedc7cf35, State: Initialized, Role: FOLLOWER
I20260924 13:44:40.302198 31971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540ddda89e104574acc48bceedc7cf35" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33465 } }
I20260924 13:44:40.302275 31971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:44:40.302304 31971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:44:40.302366 31971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:40.303017 31971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540ddda89e104574acc48bceedc7cf35" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33465 } }
I20260924 13:44:40.303134 31971 leader_election.cc:302] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 540ddda89e104574acc48bceedc7cf35; no voters:
I20260924 13:44:40.303284 31971 leader_election.cc:288] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:44:40.303610 31971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 13:44:40 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:40.305075 31976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:44:40.305229 31976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 1 LEADER]: Becoming Leader. State: Replica: 540ddda89e104574acc48bceedc7cf35, State: Running, Role: LEADER
I20260924 13:44:40.305387 31976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540ddda89e104574acc48bceedc7cf35" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33465 } }
I20260924 13:44:40.306027 31971 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 13:44:40.306001 31977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "540ddda89e104574acc48bceedc7cf35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540ddda89e104574acc48bceedc7cf35" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33465 } } }
I20260924 13:44:40.307781 31977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:40.308595 31992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:44:40.306370 31988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 540ddda89e104574acc48bceedc7cf35. Latest consensus state: current_term: 1 leader_uuid: "540ddda89e104574acc48bceedc7cf35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540ddda89e104574acc48bceedc7cf35" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33465 } } }
I20260924 13:44:40.308730 31988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:40.310036 31992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:44:40.312528 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:40.298883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.19.53.254:33465)
0924 13:44:40.299031 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:40.299041 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:40.299074 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:44:40.299537 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:40.299544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:40.299595 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:40.299739 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:40.300365 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:40.301240 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:40.302114 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:40.302696 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:40.302815 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:40.305546 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:40.305550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:40.305551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:40.305561 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:40.309685 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:40.310230 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:40.310233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:40.310233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:40.310398 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:40.310654 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:40.310655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:40.310655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:40.311194 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:40.311399 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:40.311860 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 13:44:40.312187 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260924 13:44:40.313053 31992 catalog_manager.cc:1347] Generated new cluster ID: a147b548cb22401ca1e0ebdc5bb3d7cd
I20260924 13:44:40.313112 31992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:44:40.332665 31992 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:44:40.333439 31992 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:44:40.337538 31992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35: Generated new TSK 0
I20260924 13:44:40.337786 31992 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:44:41.041846 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.53.212:51933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33465
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:44:41.154801 32017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:41.155092 32017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:41.155128 32017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:41.156766 32017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:41.156833 32017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:41.156874 32017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:44:41.158973 32017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.53.212:51933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:41.159205 32017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:41.159489 32017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:41.165720 32025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:41.165692 32022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:41.165670 32023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:41.165997 32017 server_base.cc:1034] running on GCE node
I20260924 13:44:41.166304 32017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:41.166567 32017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:41.167719 32017 hybrid_clock.cc:648] HybridClock initialized: now 1790257481167699 us; error 37 us; skew 500 ppm
Sep 24 13:44:41 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257481, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:41.170275 32017 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:44:41.170718 32017 webserver.cc:460] Webserver started at http://127.19.53.193:38903/ using document root <none> and password file <none>
I20260924 13:44:41.170943 32017 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:41.171010 32017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:41.171157 32017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:44:41 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257481, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:44:41.184856 32017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "ba6a85cd29054cfe96812198c3b7665c"
format_stamp: "Formatted at 2026-09-24 13:44:41 on dist-test-slave-tfm9"
server_key: "df2d939a2d16b5551ad3e67e3812056c"
server_key_iv: "8dc2fd1220277a215387c5d42e67ee64"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:41.185355 32017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "ba6a85cd29054cfe96812198c3b7665c"
format_stamp: "Formatted at 2026-09-24 13:44:41 on dist-test-slave-tfm9"
server_key: "df2d939a2d16b5551ad3e67e3812056c"
server_key_iv: "8dc2fd1220277a215387c5d42e67ee64"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:41.187276 32017 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260924 13:44:41.199280 32033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:41.199503 32017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:44:41.199605 32017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "ba6a85cd29054cfe96812198c3b7665c"
format_stamp: "Formatted at 2026-09-24 13:44:41 on dist-test-slave-tfm9"
server_key: "df2d939a2d16b5551ad3e67e3812056c"
server_key_iv: "8dc2fd1220277a215387c5d42e67ee64"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:41.199717 32017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:41.215178 32017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:41.225167 32017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:41.225342 32017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:41.225567 32017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:41.225948 32017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:44:41.226035 32017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:41.226091 32017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:44:41.226138 32017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:41.232709 32017 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:39731
I20260924 13:44:41.233191 32017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:44:41.233747 32148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:39731 every 8 connection(s)
I20260924 13:44:41.238849 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 32017
I20260924 13:44:41.238968 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:44:41.239163 19671 external_mini_cluster.cc:1428] Setting key f507b9b0073c9f7f30f9cc5412382f46
Sep 24 13:44:41 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257481, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:41.246517 32152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:41.234663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:38159 (local address 127.19.53.254:33465)
0924 13:44:41.234871 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:41.234873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:41.234969 (+ 96us) server_negotiation.cc:407] Connection header received
0924 13:44:41.235567 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:41.235569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:41.235583 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:41.235676 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:41.236246 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:41.236747 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:41.237809 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:41.238069 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:41.238152 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:41.242605 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:41.242607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:41.242607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:41.242614 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:41.244675 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:41.245266 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:41.245268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:41.245268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:41.245335 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:41.245656 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:41.245657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:41.245657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:41.245844 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:41.245890 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:41.246317 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 13:44:41.246409 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":137,"threads_started":1}
I20260924 13:44:41.247380 32151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:41.234614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33465 (local address 127.19.53.193:38159)
0924 13:44:41.234810 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:41.234862 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:41.235431 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:41.235756 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:41.235760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:41.235936 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:41.236129 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:41.236133 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:41.236843 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:41.236844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:41.237716 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:41.237718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:41.238174 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:41.238176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:41.238301 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:41.239468 (+ 1167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:41.239480 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:41.242497 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:41.244831 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:41.244838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:41.244841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:41.245168 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:41.245418 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:41.245418 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:41.245419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:41.245574 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:41.245966 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:41.245972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:41.246141 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 13:44:41.246892 (+ 751us) client_negotiation.cc:240] Negotiation successful
0924 13:44:41.247121 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260924 13:44:41.248330 32149 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33465
I20260924 13:44:41.248415 32149 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
I20260924 13:44:41.248600 32149 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
I20260924 13:44:41.249257 31935 ts_manager.cc:194] Registered new tserver with Master: ba6a85cd29054cfe96812198c3b7665c (127.19.53.193:39731)
I20260924 13:44:41.250188 31935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:38159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:44:41.268522 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.53.212:51933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33465
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:44:41.386822 32155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:41.387107 32155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:41.387151 32155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:41.388895 32155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:41.388950 32155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:41.388988 32155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:44:41.391139 32155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.53.212:51933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:41.391505 32155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:41.391768 32155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:41.398339 32162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:41.398368 32163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:41.398645 32165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:41.399183 32155 server_base.cc:1034] running on GCE node
I20260924 13:44:41.399387 32155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:41.399659 32155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:41.400849 32155 hybrid_clock.cc:648] HybridClock initialized: now 1790257481400835 us; error 27 us; skew 500 ppm
Sep 24 13:44:41 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790257481, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 13:44:41 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257481, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:41.403887 32155 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:44:41.404469 32155 webserver.cc:460] Webserver started at http://127.19.53.194:40225/ using document root <none> and password file <none>
I20260924 13:44:41.404736 32155 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:41.404811 32155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:41.404990 32155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:44:41 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257481, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:44:41.419678 32155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "938963bea2db460f8f8ecc174350fa74"
format_stamp: "Formatted at 2026-09-24 13:44:41 on dist-test-slave-tfm9"
server_key: "5a84687ddfee91adffcc59e915703387"
server_key_iv: "98613945a9c0296d5b33324ce5e59d5e"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:41.420275 32155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "938963bea2db460f8f8ecc174350fa74"
format_stamp: "Formatted at 2026-09-24 13:44:41 on dist-test-slave-tfm9"
server_key: "5a84687ddfee91adffcc59e915703387"
server_key_iv: "98613945a9c0296d5b33324ce5e59d5e"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:41.427416 32155 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20260924 13:44:41.440724 32172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:41.441418 32155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:41.441560 32155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "938963bea2db460f8f8ecc174350fa74"
format_stamp: "Formatted at 2026-09-24 13:44:41 on dist-test-slave-tfm9"
server_key: "5a84687ddfee91adffcc59e915703387"
server_key_iv: "98613945a9c0296d5b33324ce5e59d5e"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:41.441681 32155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:41.471026 32155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:41.473790 32155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:41.474004 32155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:41.474254 32155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:41.474661 32155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:44:41.474743 32155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:41.474803 32155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:44:41.474848 32155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:41.481184 32155 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:46425
I20260924 13:44:41.481678 32155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:44:41.484925 32288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:46425 every 8 connection(s)
I20260924 13:44:41.486601 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 32155
I20260924 13:44:41.486702 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:44:41.486908 19671 external_mini_cluster.cc:1428] Setting key 70ae4257f5c4bb87d5e673c33f5a19ad
Sep 24 13:44:41 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257481, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:41.496496 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:41.485651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33465 (local address 127.19.53.194:48097)
0924 13:44:41.485834 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:41.485889 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:41.486572 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:41.486929 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:41.486934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:41.487433 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:41.487742 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:41.487748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:41.488738 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:41.488739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:41.490061 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:41.490064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:41.490631 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:41.490632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:41.490768 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:41.491404 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:41.491413 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:41.492980 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:41.494891 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:41.494898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:41.494900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:41.495182 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:41.495411 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:41.495412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:41.495412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:41.495548 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:41.495823 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:41.495828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:41.496003 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 13:44:41.496108 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 13:44:41.496252 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260924 13:44:41.496977 32152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:41.485665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:48097 (local address 127.19.53.254:33465)
0924 13:44:41.485710 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:41.485712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:41.486003 (+ 291us) server_negotiation.cc:407] Connection header received
0924 13:44:41.486701 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:41.486703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:41.486717 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:41.486747 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:41.487855 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:41.488616 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:41.490156 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:41.490464 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:41.490553 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:41.493088 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:41.493090 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:41.493090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:41.493098 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:41.494801 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:41.495263 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:41.495264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:41.495265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:41.495334 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:41.495614 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:41.495615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:41.495615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:41.495739 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:41.495797 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:41.496799 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0924 13:44:41.496888 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:44:41.497695 32289 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33465
I20260924 13:44:41.497784 32289 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:41.497982 32289 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
I20260924 13:44:41.498343 31935 ts_manager.cc:194] Registered new tserver with Master: 938963bea2db460f8f8ecc174350fa74 (127.19.53.194:46425)
I20260924 13:44:41.498860 31935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:48097
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:44:41.528976 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.53.212:51933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33465
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:44:41.694607 32295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:41.694891 32295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:41.694924 32295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:41.696544 32295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:41.696596 32295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:41.696633 32295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:44:41.698922 32295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.53.212:51933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:41.699301 32295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:41.699571 32295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:41.706250 32303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:41.706284 32301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:41.706261 32300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:41.706720 32295 server_base.cc:1034] running on GCE node
I20260924 13:44:41.706897 32295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:41.707171 32295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:41.708348 32295 hybrid_clock.cc:648] HybridClock initialized: now 1790257481708308 us; error 60 us; skew 500 ppm
Sep 24 13:44:41 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257481, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:41.711640 32295 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:44:41.712199 32295 webserver.cc:460] Webserver started at http://127.19.53.195:40669/ using document root <none> and password file <none>
I20260924 13:44:41.712523 32295 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:41.712637 32295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:41.712834 32295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:44:41 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257481, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:44:41.734799 32295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "720dc64b7d1948d488ada576dcffa7ca"
format_stamp: "Formatted at 2026-09-24 13:44:41 on dist-test-slave-tfm9"
server_key: "b7d0ff013512018902f94c0b2e9840d5"
server_key_iv: "153216d631c80ebcf6779273f5487315"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:41.736541 32295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "720dc64b7d1948d488ada576dcffa7ca"
format_stamp: "Formatted at 2026-09-24 13:44:41 on dist-test-slave-tfm9"
server_key: "b7d0ff013512018902f94c0b2e9840d5"
server_key_iv: "153216d631c80ebcf6779273f5487315"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:41.739822 32295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 13:44:41.762814 32310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:41.763089 32295 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260924 13:44:41.763185 32295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "720dc64b7d1948d488ada576dcffa7ca"
format_stamp: "Formatted at 2026-09-24 13:44:41 on dist-test-slave-tfm9"
server_key: "b7d0ff013512018902f94c0b2e9840d5"
server_key_iv: "153216d631c80ebcf6779273f5487315"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:41.763278 32295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:41.790280 32295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:41.792831 32295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:41.793037 32295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:41.793289 32295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:41.793692 32295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:44:41.793789 32295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:41.793860 32295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:44:41.793937 32295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:41.800154 32295 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:43535
I20260924 13:44:41.800642 32295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:44:41.802626 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 32295
I20260924 13:44:41.802736 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:44:41.802955 19671 external_mini_cluster.cc:1428] Setting key 9dfad52b1f382ba328d3662104b26aff
I20260924 13:44:41.803493 32424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:43535 every 8 connection(s)
Sep 24 13:44:41 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257481, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:41.820147 32152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:41.804414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:56997 (local address 127.19.53.254:33465)
0924 13:44:41.804457 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:41.804459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:41.810192 (+ 5733us) server_negotiation.cc:407] Connection header received
0924 13:44:41.810904 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:41.810906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:41.810918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:41.810948 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:41.811801 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:41.812540 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:41.813987 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:41.814267 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:41.814308 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:41.816896 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:41.816897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:41.816898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:41.816905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:41.818624 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:41.819091 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:41.819093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:41.819093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:41.819148 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:41.819428 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:41.819429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:41.819430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:41.819553 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:41.819592 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:41.819954 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 13:44:41.820034 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:44:41.820973 32427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:41.804152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33465 (local address 127.19.53.195:56997)
0924 13:44:41.810046 (+ 5894us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:41.810099 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:41.810783 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:41.811163 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:41.811167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:41.811407 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:41.811683 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:41.811689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:41.812651 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:41.812652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:41.813859 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:41.813861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:41.814411 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:41.814412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:41.814559 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:41.815161 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:41.815172 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:41.816790 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:41.818712 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:41.818719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:41.818721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:41.819009 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:41.819231 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:41.819232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:41.819232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:41.819357 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:41.819633 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:41.819639 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:41.819807 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 13:44:41.820523 (+ 716us) client_negotiation.cc:240] Negotiation successful
0924 13:44:41.820682 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5845,"thread_start_us":80,"threads_started":1}
I20260924 13:44:41.821611 32425 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33465
I20260924 13:44:41.821759 32425 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:41.822009 32425 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
I20260924 13:44:41.822381 31935 ts_manager.cc:194] Registered new tserver with Master: 720dc64b7d1948d488ada576dcffa7ca (127.19.53.195:43535)
I20260924 13:44:41.822850 31935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:56997
I20260924 13:44:41.825280 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:44:41.826010 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 32017
I20260924 13:44:41.832830 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 32155
I20260924 13:44:41.840270 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 32295
I20260924 13:44:41.846758 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 31902
I20260924 13:44:41.853713 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33465
--webserver_interface=127.19.53.254
--webserver_port=41225
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:33465
--ranger_config_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.53.212:51933/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.53.212:51933/kms with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:44:41.938644 32428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:41.938891 32428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:41.938931 32428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:41.940124 32428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:44:41.940194 32428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:41.940218 32428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:41.940260 32428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:44:41.940284 32428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:44:41.941928 32428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.53.212:51933/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:33465
--ranger_config_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:41.942227 32428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:41.942471 32428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:41.947454 32433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:41.947463 32436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:41.947777 32428 server_base.cc:1034] running on GCE node
W20260924 13:44:41.947476 32434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:41.948184 32428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:41.948563 32428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:41.949746 32428 hybrid_clock.cc:648] HybridClock initialized: now 1790257481949733 us; error 31 us; skew 500 ppm
Sep 24 13:44:41 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257481, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:41.952819 32428 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:44:41.953292 32428 webserver.cc:460] Webserver started at http://127.19.53.254:41225/ using document root <none> and password file <none>
I20260924 13:44:41.953547 32428 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:41.953624 32428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 13:44:41 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257481, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:44:42.010821 32428 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 13:44:42.011798 32444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:42.011991 32428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:42.012102 32428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "540ddda89e104574acc48bceedc7cf35"
format_stamp: "Formatted at 2026-09-24 13:44:40 on dist-test-slave-tfm9"
server_key: "a319ad58528b220f6091c10d0c1f50ea"
server_key_iv: "3c53b24bc07e701e10f4a179648a7723"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:42.012557 32428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:42.025710 32428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:42.028249 32428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:42.028460 32428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:42.034226 32428 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:33465
I20260924 13:44:42.034257 32496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:33465 every 8 connection(s)
I20260924 13:44:42.034750 32428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:44:42.036157 32498 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:44:42.036936 32498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35: Bootstrap starting.
I20260924 13:44:42.040293 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 32428
I20260924 13:44:42.040624 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.53.212:51933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:39731
--local_ip_for_outbound_sockets=127.19.53.193
--tserver_master_addrs=127.19.53.254:33465
--webserver_port=38903
--webserver_interface=127.19.53.193
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.53.212:51933/kms with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:44:42.047518 32498 log.cc:824] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:42.050422 32498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:44:42.050915 32498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35: Bootstrap complete.
I20260924 13:44:42.055487 32498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540ddda89e104574acc48bceedc7cf35" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33465 } }
I20260924 13:44:42.055863 32498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540ddda89e104574acc48bceedc7cf35, State: Initialized, Role: FOLLOWER
I20260924 13:44:42.056123 32498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540ddda89e104574acc48bceedc7cf35" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33465 } }
I20260924 13:44:42.056207 32498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:44:42.056255 32498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:44:42.056339 32498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:44:42.057884 32498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540ddda89e104574acc48bceedc7cf35" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33465 } }
I20260924 13:44:42.058045 32498 leader_election.cc:302] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 540ddda89e104574acc48bceedc7cf35; no voters:
I20260924 13:44:42.058238 32498 leader_election.cc:288] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 13:44:42.058324 32502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:44:42.058574 32502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [term 2 LEADER]: Becoming Leader. State: Replica: 540ddda89e104574acc48bceedc7cf35, State: Running, Role: LEADER
I20260924 13:44:42.058678 32498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:44:42.058756 32502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540ddda89e104574acc48bceedc7cf35" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33465 } }
I20260924 13:44:42.059096 32502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "540ddda89e104574acc48bceedc7cf35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540ddda89e104574acc48bceedc7cf35" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33465 } } }
I20260924 13:44:42.059234 32502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:42.059422 32502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 540ddda89e104574acc48bceedc7cf35. Latest consensus state: current_term: 2 leader_uuid: "540ddda89e104574acc48bceedc7cf35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540ddda89e104574acc48bceedc7cf35" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33465 } } }
I20260924 13:44:42.059522 32502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35 [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:42.059669 32509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:44:42.060184 32509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:44:42.060401 32509 catalog_manager.cc:1259] Loaded cluster ID: a147b548cb22401ca1e0ebdc5bb3d7cd
I20260924 13:44:42.060483 32509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:44:42.060570 32498 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 13:44:42.064301 32509 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:44:42.064679 32509 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35: Loaded TSK: 0
I20260924 13:44:42.064949 32509 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:44:42.139279 32500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:42.139503 32500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:42.139537 32500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:42.140698 32500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:42.140766 32500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:42.140842 32500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:44:42.142731 32500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.53.212:51933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:39731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=38903
--tserver_master_addrs=127.19.53.254:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:42.142975 32500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:42.143270 32500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:42.149417 32533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:42.149417 32534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:42.150025 32536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:42.150566 32500 server_base.cc:1034] running on GCE node
I20260924 13:44:42.150763 32500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:42.151038 32500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:42.152196 32500 hybrid_clock.cc:648] HybridClock initialized: now 1790257482152178 us; error 40 us; skew 500 ppm
Sep 24 13:44:42 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257482, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:42.155014 32500 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:44:42.155505 32500 webserver.cc:460] Webserver started at http://127.19.53.193:38903/ using document root <none> and password file <none>
I20260924 13:44:42.155746 32500 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:42.155836 32500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 13:44:42 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257482, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:44:42.182364 32500 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.000s sys 0.001s
I20260924 13:44:42.184105 32546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:42.184340 32500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:44:42.184439 32500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "ba6a85cd29054cfe96812198c3b7665c"
format_stamp: "Formatted at 2026-09-24 13:44:41 on dist-test-slave-tfm9"
server_key: "df2d939a2d16b5551ad3e67e3812056c"
server_key_iv: "8dc2fd1220277a215387c5d42e67ee64"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:42.184985 32500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:42.204348 32500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:42.207111 32500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:42.207329 32500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:42.207620 32500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:42.208019 32500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:44:42.208096 32500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:42.208156 32500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:44:42.208204 32500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:42.216265 32500 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:39731
I20260924 13:44:42.216336 32661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:39731 every 8 connection(s)
I20260924 13:44:42.216816 32500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:44:42.217692 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 32500
I20260924 13:44:42.218092 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.53.212:51933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:46425
--local_ip_for_outbound_sockets=127.19.53.194
--tserver_master_addrs=127.19.53.254:33465
--webserver_port=40225
--webserver_interface=127.19.53.194
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.53.212:51933/kms with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:44:42 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257482, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:42.235622 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:42.218782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33465 (local address 127.19.53.193:56981)
0924 13:44:42.219626 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:42.219673 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:42.222393 (+ 2720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:42.224280 (+ 1887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:42.224284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:42.224523 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:42.224817 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:42.224824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.226677 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.226678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:42.227975 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:42.227978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.228855 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.228856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:42.228999 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:42.229667 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:42.229678 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:42.231193 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:42.233715 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:42.233721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:42.233724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:42.234045 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:42.234350 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:42.234351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:42.234351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:42.234482 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:42.234964 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:42.234971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:42.235122 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 13:44:42.235210 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 13:44:42.235375 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":98,"threads_started":1}
I20260924 13:44:42.236550 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:42.219167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:56981 (local address 127.19.53.254:33465)
0924 13:44:42.220371 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:42.220379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:42.220419 (+ 40us) server_negotiation.cc:407] Connection header received
0924 13:44:42.223921 (+ 3502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:42.223929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:42.224041 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:42.224954 (+ 913us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:42.225056 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.226555 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.228089 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.228728 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.230401 (+ 1673us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:42.231371 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:42.231376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:42.231376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:42.231389 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:42.233613 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:42.234144 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:42.234148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:42.234148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:42.234265 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:42.234564 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:42.234565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:42.234565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:42.234868 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:42.236108 (+ 1240us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:42.236200 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 13:44:42.236353 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1151,"thread_start_us":97,"threads_started":1}
I20260924 13:44:42.237622 32662 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33465
I20260924 13:44:42.237737 32662 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:42.237996 32662 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
W20260924 13:44:42.239012 32662 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:33465 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 13:44:42.239100 32662 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:42.239169 32662 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
I20260924 13:44:42.239684 32662 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:42.239786 32662 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
W20260924 13:44:42.240209 32662 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:44:42.352860 32666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:42.353149 32666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:42.353191 32666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:42.354913 32666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:42.354975 32666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:42.355017 32666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:44:42.357185 32666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.53.212:51933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:46425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=40225
--tserver_master_addrs=127.19.53.254:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:42.357590 32666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:42.357867 32666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:42.363648 32675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:42.363654 32672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:42.364246 32673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:42.364859 32666 server_base.cc:1034] running on GCE node
I20260924 13:44:42.365069 32666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:42.365337 32666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:42.366504 32666 hybrid_clock.cc:648] HybridClock initialized: now 1790257482366485 us; error 32 us; skew 500 ppm
Sep 24 13:44:42 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257482, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:42.369771 32666 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:44:42.370306 32666 webserver.cc:460] Webserver started at http://127.19.53.194:40225/ using document root <none> and password file <none>
I20260924 13:44:42.370671 32666 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:42.370803 32666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 13:44:42 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257482, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:44:42.386713 32666 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:44:42.387871 32683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:42.388118 32666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:44:42.388489 32666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "938963bea2db460f8f8ecc174350fa74"
format_stamp: "Formatted at 2026-09-24 13:44:41 on dist-test-slave-tfm9"
server_key: "5a84687ddfee91adffcc59e915703387"
server_key_iv: "98613945a9c0296d5b33324ce5e59d5e"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:42.388969 32666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:42.416597 32666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:42.419163 32666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:42.419358 32666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:42.419636 32666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:42.420071 32666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:44:42.420174 32666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:42.420274 32666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:44:42.420338 32666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:42.428736 32666 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:46425
I20260924 13:44:42.428913 329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:46425 every 8 connection(s)
I20260924 13:44:42.429246 32666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:44:42.432173 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 32666
I20260924 13:44:42.432471 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.53.212:51933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:43535
--local_ip_for_outbound_sockets=127.19.53.195
--tserver_master_addrs=127.19.53.254:33465
--webserver_port=40669
--webserver_interface=127.19.53.195
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.53.212:51933/kms with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:44:42 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257482, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:42.447556 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:42.430068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:52805 (local address 127.19.53.254:33465)
0924 13:44:42.430125 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:42.430127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:42.430423 (+ 296us) server_negotiation.cc:407] Connection header received
0924 13:44:42.431128 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:42.431129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:42.431147 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:42.431221 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:42.431897 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.432683 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.434271 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.434499 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.434579 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:42.443832 (+ 9253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:42.443835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:42.443835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:42.443847 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:42.445721 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:42.446326 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:42.446327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:42.446328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:42.446398 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:42.446705 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:42.446706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:42.446706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:42.446860 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:42.446918 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:42.447307 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 13:44:42.447406 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:44:42.448546 332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:42.430095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33465 (local address 127.19.53.194:52805)
0924 13:44:42.430280 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:42.430330 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:42.430991 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:42.431298 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:42.431302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:42.431530 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:42.431766 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:42.431778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.432818 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.432819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:42.434163 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:42.434168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.434607 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.434608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:42.434746 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:42.442046 (+ 7300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:42.442057 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:42.443671 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:42.445840 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:42.445849 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:42.445851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:42.446222 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:42.446481 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:42.446481 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:42.446482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:42.446620 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:42.446951 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:42.446957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:42.447129 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 13:44:42.448079 (+ 950us) client_negotiation.cc:240] Negotiation successful
0924 13:44:42.448287 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260924 13:44:42.449249 330 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33465
I20260924 13:44:42.449349 330 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:42.449542 330 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
W20260924 13:44:42.450254 330 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:33465 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 13:44:42.450326 330 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:42.450397 330 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
I20260924 13:44:42.450830 330 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:42.450909 330 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
W20260924 13:44:42.451275 330 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:44:42.541090 333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:42.541373 333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:42.541414 333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:42.543104 333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:42.543171 333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:42.543215 333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:44:42.545416 333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.53.212:51933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:43535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=40669
--tserver_master_addrs=127.19.53.254:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:42.545838 333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:42.546154 333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:42.553308 341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:42.553567 339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:42.554088 338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:42.554234 333 server_base.cc:1034] running on GCE node
I20260924 13:44:42.554548 333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:42.554880 333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:42.556084 333 hybrid_clock.cc:648] HybridClock initialized: now 1790257482556059 us; error 38 us; skew 500 ppm
Sep 24 13:44:42 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257482, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:42.559585 333 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:44:42.560271 333 webserver.cc:460] Webserver started at http://127.19.53.195:40669/ using document root <none> and password file <none>
I20260924 13:44:42.560638 333 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:42.560757 333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 13:44:42 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257482, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:44:42.586331 333 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 13:44:42.587422 350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:42.587613 333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:42.587704 333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "720dc64b7d1948d488ada576dcffa7ca"
format_stamp: "Formatted at 2026-09-24 13:44:41 on dist-test-slave-tfm9"
server_key: "b7d0ff013512018902f94c0b2e9840d5"
server_key_iv: "153216d631c80ebcf6779273f5487315"
server_key_version: "kuduclusterkey@0"
I20260924 13:44:42.588207 333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:42.617671 333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:42.619803 333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:42.619997 333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:42.620288 333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:42.620746 333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:44:42.620805 333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:42.620855 333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:44:42.620884 333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:42.628895 333 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:43535
I20260924 13:44:42.629417 333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:44:42.630648 463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:43535 every 8 connection(s)
I20260924 13:44:42.632247 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 333
Sep 24 13:44:42 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257482, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:42.642439 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:42.631373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33465 (local address 127.19.53.195:35585)
0924 13:44:42.631752 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:42.631795 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:42.632417 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:42.633187 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:42.633191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:42.633435 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:42.633720 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:42.633725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.634720 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.634721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:42.635965 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:42.635968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.636438 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.636440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:42.636568 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:42.637201 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:42.637211 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:42.638879 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:42.640791 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:42.640798 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:42.640800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:42.641089 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:42.641321 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:42.641322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:42.641323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:42.641455 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:42.641760 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:42.641766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:42.641952 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 13:44:42.642067 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 13:44:42.642222 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":78,"threads_started":1}
I20260924 13:44:42.642882 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:42.631385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:35585 (local address 127.19.53.254:33465)
0924 13:44:42.631433 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:42.631435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:42.631904 (+ 469us) server_negotiation.cc:407] Connection header received
0924 13:44:42.632957 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:42.632958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:42.632970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:42.633035 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:42.633840 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.634607 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.636053 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.636307 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.636404 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:42.638998 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:42.639000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:42.639000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:42.639007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:42.640679 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:42.641175 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:42.641176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:42.641177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:42.641234 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:42.641528 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:42.641529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:42.641530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:42.641666 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:42.641731 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:42.642715 (+ 984us) server_negotiation.cc:299] Negotiation successful
0924 13:44:42.642798 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":23}
I20260924 13:44:42.643006 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:42.632854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.19.53.254:33465)
0924 13:44:42.633020 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:42.633022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:42.633030 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:44:42.633053 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:42.633054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:42.633066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:42.633984 (+ 918us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:42.634023 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.634744 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.635671 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:42.635864 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:42.635952 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:42.637485 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:42.637487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:42.637487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:42.637494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:42.639194 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:42.639777 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:42.639778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:42.639779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:42.639843 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:42.642009 (+ 2166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:42.642010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:42.642010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:42.642158 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:42.642345 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:42.642801 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 13:44:42.642899 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":59,"threads_started":1}
I20260924 13:44:42.643220 464 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33465
I20260924 13:44:42.643337 464 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:42.643530 464 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
W20260924 13:44:42.644402 464 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:33465 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 13:44:42.644469 464 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:42.644513 464 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
I20260924 13:44:42.644846 464 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:42.644927 464 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
W20260924 13:44:42.645195 464 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 13:44:43 dist-test-slave-tfm9 krb5kdc[31450](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790257483, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:43.240422 32662 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:43.240595 32662 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
I20260924 13:44:43.241021 32461 ts_manager.cc:194] Registered new tserver with Master: ba6a85cd29054cfe96812198c3b7665c (127.19.53.193:39731)
I20260924 13:44:43.242111 32461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:56981
I20260924 13:44:43.451465 330 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:43.451615 330 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
I20260924 13:44:43.452014 32461 ts_manager.cc:194] Registered new tserver with Master: 938963bea2db460f8f8ecc174350fa74 (127.19.53.194:46425)
I20260924 13:44:43.452616 32461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:52805
Sep 24 13:44:43 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257483, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
I20260924 13:44:43.645380 464 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:43.645529 464 heartbeater.cc:511] Master 127.19.53.254:33465 requested a full tablet report, sending...
I20260924 13:44:43.645857 32461 ts_manager.cc:194] Registered new tserver with Master: 720dc64b7d1948d488ada576dcffa7ca (127.19.53.195:43535)
I20260924 13:44:43.646386 32461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:35585
I20260924 13:44:43.651747 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:44:43.677762 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:43.658199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40576 (local address 127.19.53.254:33465)
0924 13:44:43.660044 (+ 1845us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:43.660046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:43.661297 (+ 1251us) server_negotiation.cc:407] Connection header received
0924 13:44:43.661326 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:43.661326 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:43.661340 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:43.661400 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:43.662176 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.662974 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.664121 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.664451 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.664515 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:43.674383 (+ 9868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:43.674384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:43.674385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:43.674394 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:43.676209 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:43.676612 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:43.676613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:43.676614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:43.676670 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:43.676854 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:43.676854 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:43.676855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:43.677023 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:43.677116 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:43.677571 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 13:44:43.677654 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1817,"thread_start_us":85,"threads_started":1}
I20260924 13:44:43.680137 32461 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:44:43.682499 32461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:44:43.702256 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:43.686206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:46425 (local address 127.0.0.1:53634)
0924 13:44:43.687172 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:43.687182 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:43.687230 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:43.695414 (+ 8184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:43.695415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:43.695418 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:43.695474 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:43.695475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.696663 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.696663 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:43.698256 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:43.698260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.701866 (+ 3606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.701867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:43.701992 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:43.702013 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 13:44:43.702097 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 13:44:43.702157 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":64,"threads_started":1}
I20260924 13:44:43.702857 494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:43.686311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.19.53.194:46425)
0924 13:44:43.693758 (+ 7447us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:43.693763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:43.693774 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:44:43.693810 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:43.693812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:43.693827 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:43.693902 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:43.695566 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.696560 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.698373 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.699178 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.699273 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:43.699338 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:43.702720 (+ 3382us) server_negotiation.cc:299] Negotiation successful
0924 13:44:43.702771 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7404,"thread_start_us":44,"threads_started":1}
I20260924 13:44:43.703771 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:43.686593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58314 (local address 127.19.53.193:39731)
0924 13:44:43.687114 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:43.687120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:43.693704 (+ 6584us) server_negotiation.cc:407] Connection header received
0924 13:44:43.693751 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:43.693754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:43.693774 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:43.693863 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:43.694204 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.695313 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.700523 (+ 5210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.701672 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.701766 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:43.701826 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:43.703649 (+ 1823us) server_negotiation.cc:299] Negotiation successful
0924 13:44:43.703695 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":72,"threads_started":1}
I20260924 13:44:43.703564 491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:43.686386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39731 (local address 127.0.0.1:58314)
0924 13:44:43.693540 (+ 7154us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:43.693568 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:43.693628 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:43.694010 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:43.694013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:43.694016 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:43.694103 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:43.694105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.697125 (+ 3020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.697126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:43.700403 (+ 3277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:43.700406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.703320 (+ 2914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.703321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:43.703386 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:43.703391 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:44:43.703451 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 13:44:43.703491 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7134,"mutex_wait_us":2147,"thread_start_us":75,"threads_started":1}
I20260924 13:44:43.704800 32713 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3f712de2b5420380ed5aad3084ce24 (DEFAULT_TABLE table=test-table [id=2728aca03e994d52ae30f39ce4d04d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:44:43.705051 32589 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3f712de2b5420380ed5aad3084ce24 (DEFAULT_TABLE table=test-table [id=2728aca03e994d52ae30f39ce4d04d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:44:43.705173 32713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3f712de2b5420380ed5aad3084ce24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:43.705350 32589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3f712de2b5420380ed5aad3084ce24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:43.709666 499 tablet_bootstrap.cc:492] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c: Bootstrap starting.
I20260924 13:44:43.710119 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:43.685902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:43535 (local address 127.0.0.1:49554)
0924 13:44:43.686817 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:43.686859 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:43.686967 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:43.694326 (+ 7359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:43.694327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:43.694329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:43.694366 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:43.694367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.695598 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.695599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:43.696800 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:43.696803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.709843 (+ 13040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.709844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:43.709923 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:43.709927 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:44:43.709984 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 13:44:43.710041 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":74,"threads_started":1}
I20260924 13:44:43.710453 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:43.686009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49554 (local address 127.19.53.195:43535)
0924 13:44:43.694066 (+ 8057us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:43.694070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:43.694082 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:44:43.694115 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:43.694117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:43.694133 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:43.694228 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:43.694454 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.695500 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.708865 (+ 13365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.709652 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.709738 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:43.709800 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:43.710296 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 13:44:43.710353 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8021,"thread_start_us":61,"threads_started":1}
I20260924 13:44:43.710676 499 tablet_bootstrap.cc:654] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:43.711026 499 log.cc:824] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:43.711146 398 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3f712de2b5420380ed5aad3084ce24 (DEFAULT_TABLE table=test-table [id=2728aca03e994d52ae30f39ce4d04d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:44:43.711395 498 tablet_bootstrap.cc:492] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74: Bootstrap starting.
I20260924 13:44:43.711491 398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3f712de2b5420380ed5aad3084ce24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:43.711968 498 tablet_bootstrap.cc:654] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:43.712272 498 log.cc:824] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:43.712981 499 tablet_bootstrap.cc:492] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c: No bootstrap required, opened a new log
I20260924 13:44:43.713053 499 ts_tablet_manager.cc:1397] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 13:44:43.714745 502 tablet_bootstrap.cc:492] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca: Bootstrap starting.
I20260924 13:44:43.715294 502 tablet_bootstrap.cc:654] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:43.715646 502 log.cc:824] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:43.719753 499 raft_consensus.cc:348] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dc64b7d1948d488ada576dcffa7ca" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43535 } } peers { permanent_uuid: "938963bea2db460f8f8ecc174350fa74" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46425 } } peers { permanent_uuid: "ba6a85cd29054cfe96812198c3b7665c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39731 } }
I20260924 13:44:43.720362 498 tablet_bootstrap.cc:492] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74: No bootstrap required, opened a new log
I20260924 13:44:43.720474 498 ts_tablet_manager.cc:1397] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 13:44:43.722949 498 raft_consensus.cc:348] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dc64b7d1948d488ada576dcffa7ca" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43535 } } peers { permanent_uuid: "938963bea2db460f8f8ecc174350fa74" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46425 } } peers { permanent_uuid: "ba6a85cd29054cfe96812198c3b7665c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39731 } }
I20260924 13:44:43.723110 498 raft_consensus.cc:374] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:43.723140 498 raft_consensus.cc:729] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938963bea2db460f8f8ecc174350fa74, State: Initialized, Role: FOLLOWER
I20260924 13:44:43.723253 498 consensus_queue.cc:260] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dc64b7d1948d488ada576dcffa7ca" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43535 } } peers { permanent_uuid: "938963bea2db460f8f8ecc174350fa74" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46425 } } peers { permanent_uuid: "ba6a85cd29054cfe96812198c3b7665c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39731 } }
I20260924 13:44:43.723498 498 ts_tablet_manager.cc:1428] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:44:43.723620 499 raft_consensus.cc:374] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:43.723634 330 heartbeater.cc:503] Master 127.19.53.254:33465 was elected leader, sending a full tablet report...
I20260924 13:44:43.723677 499 raft_consensus.cc:729] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba6a85cd29054cfe96812198c3b7665c, State: Initialized, Role: FOLLOWER
I20260924 13:44:43.723799 499 consensus_queue.cc:260] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dc64b7d1948d488ada576dcffa7ca" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43535 } } peers { permanent_uuid: "938963bea2db460f8f8ecc174350fa74" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46425 } } peers { permanent_uuid: "ba6a85cd29054cfe96812198c3b7665c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39731 } }
I20260924 13:44:43.724051 499 ts_tablet_manager.cc:1428] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20260924 13:44:43.724352 32662 heartbeater.cc:503] Master 127.19.53.254:33465 was elected leader, sending a full tablet report...
I20260924 13:44:43.724783 502 tablet_bootstrap.cc:492] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca: No bootstrap required, opened a new log
I20260924 13:44:43.724849 502 ts_tablet_manager.cc:1397] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260924 13:44:43.727334 502 raft_consensus.cc:348] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dc64b7d1948d488ada576dcffa7ca" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43535 } } peers { permanent_uuid: "938963bea2db460f8f8ecc174350fa74" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46425 } } peers { permanent_uuid: "ba6a85cd29054cfe96812198c3b7665c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39731 } }
I20260924 13:44:43.727514 502 raft_consensus.cc:374] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:43.727550 502 raft_consensus.cc:729] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720dc64b7d1948d488ada576dcffa7ca, State: Initialized, Role: FOLLOWER
I20260924 13:44:43.727658 502 consensus_queue.cc:260] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dc64b7d1948d488ada576dcffa7ca" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43535 } } peers { permanent_uuid: "938963bea2db460f8f8ecc174350fa74" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46425 } } peers { permanent_uuid: "ba6a85cd29054cfe96812198c3b7665c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39731 } }
I20260924 13:44:43.727917 502 ts_tablet_manager.cc:1428] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:44:43.727952 464 heartbeater.cc:503] Master 127.19.53.254:33465 was elected leader, sending a full tablet report...
Sep 24 13:44:43 dist-test-slave-tfm9 krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257483, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for HTTP/127.19.53.212@KRBTEST.COM
W20260924 13:44:43.882464 465 tablet.cc:2369] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:44:43.895777 506 raft_consensus.cc:482] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:44:43.895926 506 raft_consensus.cc:504] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dc64b7d1948d488ada576dcffa7ca" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43535 } } peers { permanent_uuid: "938963bea2db460f8f8ecc174350fa74" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46425 } } peers { permanent_uuid: "ba6a85cd29054cfe96812198c3b7665c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39731 } }
I20260924 13:44:43.896328 506 leader_election.cc:288] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 938963bea2db460f8f8ecc174350fa74 (127.19.53.194:46425), ba6a85cd29054cfe96812198c3b7665c (127.19.53.193:39731)
I20260924 13:44:43.904357 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:43.896718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:48547 (local address 127.19.53.193:39731)
0924 13:44:43.896766 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:43.896768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:43.897222 (+ 454us) server_negotiation.cc:407] Connection header received
0924 13:44:43.897362 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:43.897363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:43.897374 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:43.897404 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:43.898897 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.899673 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.903145 (+ 3472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.903747 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.903797 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:43.903806 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:43.904221 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 13:44:43.904268 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:44:43.904772 509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:43.896486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39731 (local address 127.19.53.195:48547)
0924 13:44:43.897163 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:43.897175 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:43.897253 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:43.897723 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:43.897724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:43.897727 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:43.898799 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:43.898805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.899901 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.899902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:43.903041 (+ 3139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:43.903044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.904050 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.904051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:43.904122 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:43.904126 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:44:43.904624 (+ 498us) client_negotiation.cc:240] Negotiation successful
0924 13:44:43.904670 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":98,"threads_started":1}
I20260924 13:44:43.905388 32607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f712de2b5420380ed5aad3084ce24" candidate_uuid: "720dc64b7d1948d488ada576dcffa7ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba6a85cd29054cfe96812198c3b7665c" is_pre_election: true
I20260924 13:44:43.905661 32607 raft_consensus.cc:2454] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720dc64b7d1948d488ada576dcffa7ca in term 0.
I20260924 13:44:43.905947 351 leader_election.cc:302] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720dc64b7d1948d488ada576dcffa7ca, ba6a85cd29054cfe96812198c3b7665c; no voters:
I20260924 13:44:43.906104 506 raft_consensus.cc:2781] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:44:43.906157 506 raft_consensus.cc:482] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:44:43.906186 506 raft_consensus.cc:3037] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:43.907065 506 raft_consensus.cc:504] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dc64b7d1948d488ada576dcffa7ca" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43535 } } peers { permanent_uuid: "938963bea2db460f8f8ecc174350fa74" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46425 } } peers { permanent_uuid: "ba6a85cd29054cfe96812198c3b7665c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39731 } }
I20260924 13:44:43.907194 506 leader_election.cc:288] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [CANDIDATE]: Term 1 election: Requested vote from peers 938963bea2db460f8f8ecc174350fa74 (127.19.53.194:46425), ba6a85cd29054cfe96812198c3b7665c (127.19.53.193:39731)
I20260924 13:44:43.907541 32607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f712de2b5420380ed5aad3084ce24" candidate_uuid: "720dc64b7d1948d488ada576dcffa7ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba6a85cd29054cfe96812198c3b7665c"
I20260924 13:44:43.907608 32607 raft_consensus.cc:3037] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:43.908452 32607 raft_consensus.cc:2454] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dc64b7d1948d488ada576dcffa7ca in term 1.
I20260924 13:44:43.908720 351 leader_election.cc:302] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720dc64b7d1948d488ada576dcffa7ca, ba6a85cd29054cfe96812198c3b7665c; no voters:
I20260924 13:44:43.908839 506 raft_consensus.cc:2781] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:44:43.908982 506 raft_consensus.cc:686] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [term 1 LEADER]: Becoming Leader. State: Replica: 720dc64b7d1948d488ada576dcffa7ca, State: Running, Role: LEADER
I20260924 13:44:43.909075 506 consensus_queue.cc:237] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dc64b7d1948d488ada576dcffa7ca" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43535 } } peers { permanent_uuid: "938963bea2db460f8f8ecc174350fa74" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46425 } } peers { permanent_uuid: "ba6a85cd29054cfe96812198c3b7665c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39731 } }
I20260924 13:44:43.909984 32460 catalog_manager.cc:5495] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 720dc64b7d1948d488ada576dcffa7ca (127.19.53.195). New cstate: current_term: 1 leader_uuid: "720dc64b7d1948d488ada576dcffa7ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dc64b7d1948d488ada576dcffa7ca" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938963bea2db460f8f8ecc174350fa74" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba6a85cd29054cfe96812198c3b7665c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39731 } health_report { overall_health: UNKNOWN } } }
I20260924 13:44:43.913486 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:43.896767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:46425 (local address 127.19.53.195:45853)
0924 13:44:43.901972 (+ 5205us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:43.901984 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:43.902067 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:43.902239 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:43.902240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:43.902241 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:43.902323 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:43.902324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.905799 (+ 3475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.905800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:43.906913 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:43.906916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.913214 (+ 6298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.913216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:43.913300 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:43.913305 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:44:43.913362 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 13:44:43.913403 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5186,"thread_start_us":65,"threads_started":1}
I20260924 13:44:43.913781 494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:43.896957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:45853 (local address 127.19.53.194:46425)
0924 13:44:43.896994 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:43.896995 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:43.902041 (+ 5046us) server_negotiation.cc:407] Connection header received
0924 13:44:43.902138 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:43.902139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:43.902149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:43.902177 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:43.902405 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.903077 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.911393 (+ 8316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.913115 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.913191 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:43.913200 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:43.913664 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 13:44:43.913714 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:44:43.914199 32751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f712de2b5420380ed5aad3084ce24" candidate_uuid: "720dc64b7d1948d488ada576dcffa7ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938963bea2db460f8f8ecc174350fa74" is_pre_election: true
I20260924 13:44:43.914355 32751 raft_consensus.cc:2454] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720dc64b7d1948d488ada576dcffa7ca in term 0.
I20260924 13:44:43.914484 32750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f712de2b5420380ed5aad3084ce24" candidate_uuid: "720dc64b7d1948d488ada576dcffa7ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938963bea2db460f8f8ecc174350fa74"
I20260924 13:44:43.914543 32750 raft_consensus.cc:3037] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:43.915465 32750 raft_consensus.cc:2454] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dc64b7d1948d488ada576dcffa7ca in term 1.
I20260924 13:44:43.930146 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:43.926666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.19.53.195:43535)
0924 13:44:43.926723 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:43.926725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:43.926732 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:44:43.926772 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:43.926773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:43.926794 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:43.926873 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:44:43.927059 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.927831 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.928990 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:43.929313 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:43.929384 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:43.929597 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:44:43.929737 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:44:43.929819 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:43.930046 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 13:44:43.930068 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 13:44:43.933342 331 tablet.cc:2369] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:44:43.933382 32607 raft_consensus.cc:1261] T 4d3f712de2b5420380ed5aad3084ce24 P ba6a85cd29054cfe96812198c3b7665c [term 1 FOLLOWER]: Refusing update from remote peer 720dc64b7d1948d488ada576dcffa7ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:44:43.933720 510 consensus_queue.cc:1035] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba6a85cd29054cfe96812198c3b7665c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:44:43.933847 32750 raft_consensus.cc:1261] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74 [term 1 FOLLOWER]: Refusing update from remote peer 720dc64b7d1948d488ada576dcffa7ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:44:43.934116 510 consensus_queue.cc:1035] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "938963bea2db460f8f8ecc174350fa74" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:44:43.936800 513 mvcc.cc:205] Tried to move back new op lower bound from 7332894654185562112 to 7332894654092111872. Current Snapshot: MvccSnapshot[applied={T|T < 7332894654185562112}]
I20260924 13:44:43.937261 514 mvcc.cc:205] Tried to move back new op lower bound from 7332894654185562112 to 7332894654092111872. Current Snapshot: MvccSnapshot[applied={T|T < 7332894654185562112}]
I20260924 13:44:43.941879 32460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:44:43.942019 32460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:44:43.943176 32460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 540ddda89e104574acc48bceedc7cf35: Sending DeleteTablet for 3 replicas of tablet 4d3f712de2b5420380ed5aad3084ce24
I20260924 13:44:43.943594 32713 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3f712de2b5420380ed5aad3084ce24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:44:43 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:53634
I20260924 13:44:43.943732 398 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3f712de2b5420380ed5aad3084ce24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:44:43 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:49554
I20260924 13:44:43.945024 516 mvcc.cc:205] Tried to move back new op lower bound from 7332894654185562112 to 7332894654092111872. Current Snapshot: MvccSnapshot[applied={T|T < 7332894654185562112}]
I20260924 13:44:43.945359 524 tablet_replica.cc:330] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca: stopping tablet replica
I20260924 13:44:43.945504 524 raft_consensus.cc:2229] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:44:43.945638 524 raft_consensus.cc:2258] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:44:43.945850 523 tablet_replica.cc:330] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74: stopping tablet replica
I20260924 13:44:43.946161 32589 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3f712de2b5420380ed5aad3084ce24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:44:43 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:58314
I20260924 13:44:43.946357 524 ts_tablet_manager.cc:1905] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:44:43.946707 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 32500
I20260924 13:44:43.947291 523 raft_consensus.cc:2229] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:44:43.947403 523 raft_consensus.cc:2258] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:44:43.949121 523 ts_tablet_manager.cc:1905] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:44:43.950323 523 ts_tablet_manager.cc:1918] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:44:43.950419 523 log.cc:1196] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/4d3f712de2b5420380ed5aad3084ce24
I20260924 13:44:43.950676 523 ts_tablet_manager.cc:1939] T 4d3f712de2b5420380ed5aad3084ce24 P 938963bea2db460f8f8ecc174350fa74: Deleting consensus metadata
I20260924 13:44:43.951089 524 ts_tablet_manager.cc:1918] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:44:43.951113 32447 catalog_manager.cc:4841] TS 938963bea2db460f8f8ecc174350fa74 (127.19.53.194:46425): tablet 4d3f712de2b5420380ed5aad3084ce24 (table test-table [id=2728aca03e994d52ae30f39ce4d04d40]) successfully deleted
I20260924 13:44:43.951222 524 log.cc:1196] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/4d3f712de2b5420380ed5aad3084ce24
I20260924 13:44:43.951495 524 ts_tablet_manager.cc:1939] T 4d3f712de2b5420380ed5aad3084ce24 P 720dc64b7d1948d488ada576dcffa7ca: Deleting consensus metadata
I20260924 13:44:43.951892 32448 catalog_manager.cc:4841] TS 720dc64b7d1948d488ada576dcffa7ca (127.19.53.195:43535): tablet 4d3f712de2b5420380ed5aad3084ce24 (table test-table [id=2728aca03e994d52ae30f39ce4d04d40]) successfully deleted
W20260924 13:44:43.965519 32445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:39731 (error 108)
I20260924 13:44:43.974138 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 32666
I20260924 13:44:43.976537 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:43.976375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39731 (local address 127.0.0.1:58330)
0924 13:44:43.976430 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:43.976488 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 13:44:43.976698 32445 catalog_manager.cc:4584] TS ba6a85cd29054cfe96812198c3b7665c (127.19.53.193:39731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d3f712de2b5420380ed5aad3084ce24: Network error: Client connection negotiation failed: client connection to 127.19.53.193:39731: connect: Connection refused (error 111)
I20260924 13:44:43.982105 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 333
I20260924 13:44:43.987589 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 32428
I20260924 13:44:43.997804 19671 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 13:44:44.342108 19671 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 13:44:44.342222 19671 mini_ranger.cc:66] Stopping Ranger...
I20260924 13:44:44.686445 19671 mini_ranger.cc:68] Stopped Ranger
2026-09-24 13:44:44.686 UTC [31480] LOG: received smart shutdown request
2026-09-24 13:44:44.690 UTC [31480] LOG: background worker "logical replication launcher" (PID 31489) exited with exit code 1
2026-09-24 13:44:44.690 UTC [31484] LOG: shutting down
2026-09-24 13:44:44.863 UTC [31480] LOG: database system is shut down
2026-09-24T13:44:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47844 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:44:45 dist-test-slave-tfm9 krb5kdc[536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:44:45 dist-test-slave-tfm9 krb5kdc[536](info): set up 2 sockets
Sep 24 13:44:45 dist-test-slave-tfm9 krb5kdc[536](info): commencing operation
krb5kdc: starting...
W20260924 13:44:46.007586 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:44:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:44:46.123011 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:43947
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:43947,127.19.53.253:35141,127.19.53.252:44717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:44:46.201642 549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:46.201879 549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:46.201916 549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:46.203049 549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:44:46.203114 549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:46.203138 549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:46.203167 549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:44:46.203197 549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:44:46.204670 549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:43947,127.19.53.253:35141,127.19.53.252:44717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:43947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:46.204934 549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:46.205152 549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:46.209980 554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:46.210026 557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:46.210035 555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:46.210156 549 server_base.cc:1034] running on GCE node
I20260924 13:44:46.210419 549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:46.210695 549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:46.211880 549 hybrid_clock.cc:648] HybridClock initialized: now 1790257486211861 us; error 36 us; skew 500 ppm
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:46.214414 549 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:44:46.214856 549 webserver.cc:460] Webserver started at http://127.19.53.254:46711/ using document root <none> and password file <none>
I20260924 13:44:46.215066 549 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:46.215106 549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:46.215188 549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:44:46.216135 549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "1c3612329f154f40a59f74f712d70603"
format_stamp: "Formatted at 2026-09-24 13:44:46 on dist-test-slave-tfm9"
server_key: "5080564d9b04cf4f0731ed3d03b900d2"
server_key_iv: "381ef2edb1946980b8567424888599d6"
server_key_version: "encryptionkey@0"
I20260924 13:44:46.216408 549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "1c3612329f154f40a59f74f712d70603"
format_stamp: "Formatted at 2026-09-24 13:44:46 on dist-test-slave-tfm9"
server_key: "5080564d9b04cf4f0731ed3d03b900d2"
server_key_iv: "381ef2edb1946980b8567424888599d6"
server_key_version: "encryptionkey@0"
I20260924 13:44:46.217798 549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:46.218519 564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:46.218689 549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:46.218801 549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "1c3612329f154f40a59f74f712d70603"
format_stamp: "Formatted at 2026-09-24 13:44:46 on dist-test-slave-tfm9"
server_key: "5080564d9b04cf4f0731ed3d03b900d2"
server_key_iv: "381ef2edb1946980b8567424888599d6"
server_key_version: "encryptionkey@0"
I20260924 13:44:46.218878 549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:46.228868 549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:46.231110 549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:46.231269 549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:46.235299 549 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:43947
I20260924 13:44:46.235332 617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:43947 every 8 connection(s)
I20260924 13:44:46.235666 549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:44:46.236511 618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:46.237341 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 549
I20260924 13:44:46.237468 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:44:46.237660 19671 external_mini_cluster.cc:1428] Setting key 7aaa7c67b12ee5652d1bc71729932af8
I20260924 13:44:46.238612 618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.53.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.253@KRBTEST.COM" created.
I20260924 13:44:46.248131 622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.239494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.19.53.254:43947)
0924 13:44:46.239681 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.239691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.239802 (+ 111us) server_negotiation.cc:407] Connection header received
0924 13:44:46.240333 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.240336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.240355 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.240462 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.240978 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.241844 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.242733 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.243065 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.243192 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.245141 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.245147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.245148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.245154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.246588 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.246997 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.246999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.246999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.247074 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.247308 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.247309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.247309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.247591 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.247681 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.247882 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.247981 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":100,"threads_started":1}
I20260924 13:44:46.248255 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.239473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:43947 (local address 127.0.0.1:56862)
0924 13:44:46.239647 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.239689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:46.240176 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:46.240467 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:46.240470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:46.240737 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:46.240881 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.240885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.241956 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.241957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.242633 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.242635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.243164 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.243165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.243228 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.243743 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:46.243751 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:46.245048 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:46.246690 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.246696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.246698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.246901 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.247141 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.247142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.247142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.247227 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.247660 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:46.247665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:46.247728 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 13:44:46.247836 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 13:44:46.248061 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":64,"thread_start_us":83,"threads_started":1}
I20260924 13:44:46.249151 618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:44:46.249495 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.249377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:35141 (local address 127.0.0.1:57432)
0924 13:44:46.249425 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.249465 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 13:44:46.249660 565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.53.253:35141: connect: Connection refused (error 111)
I20260924 13:44:46.250104 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.250016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:35141 (local address 127.0.0.1:57438)
0924 13:44:46.250059 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.250083 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 13:44:46.250207 618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.253:35141: Network error: Client connection negotiation failed: client connection to 127.19.53.253:35141: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.253.keytab.
Entry for principal kudu/127.19.53.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.253.keytab.
I20260924 13:44:46.262351 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.253:35141
--webserver_interface=127.19.53.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.253.keytab
--principal=kudu/127.19.53.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:43947,127.19.53.253:35141,127.19.53.252:44717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:44:46.299441 618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } attempt: 1
I20260924 13:44:46.299983 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.299836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:35141 (local address 127.0.0.1:57446)
0924 13:44:46.299907 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.299951 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 13:44:46.300369 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.300287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:35141 (local address 127.0.0.1:57456)
0924 13:44:46.300327 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.300351 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 13:44:46.300472 618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.253:35141: Network error: Client connection negotiation failed: client connection to 127.19.53.253:35141: connect: Connection refused (error 111)
W20260924 13:44:46.342921 625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:46.343118 625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:46.343142 625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:46.344246 625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:44:46.344288 625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:46.344302 625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:46.344314 625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:44:46.344362 625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:44:46.345854 625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:43947,127.19.53.253:35141,127.19.53.252:44717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.253.keytab
--principal=kudu/127.19.53.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.253:35141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.53.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:46.346122 625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:46.346354 625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:46.351181 633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:46.351155 631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:46.351125 630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:46.351231 625 server_base.cc:1034] running on GCE node
I20260924 13:44:46.351568 625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:46.351838 625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:46.353047 625 hybrid_clock.cc:648] HybridClock initialized: now 1790257486353026 us; error 40 us; skew 500 ppm
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:46.356005 625 init.cc:377] Logged in from keytab as kudu/127.19.53.253@KRBTEST.COM (short username kudu)
I20260924 13:44:46.356459 625 webserver.cc:460] Webserver started at http://127.19.53.253:39643/ using document root <none> and password file <none>
I20260924 13:44:46.356694 625 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:46.356770 625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:46.356918 625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:44:46.358002 625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/data/instance:
uuid: "c2a41ac0b004449b87463ed997467a04"
format_stamp: "Formatted at 2026-09-24 13:44:46 on dist-test-slave-tfm9"
server_key: "2936e99f247b06acfd8c2608917cdd1a"
server_key_iv: "778a546ec7ab0bdb6ccabe21df611e37"
server_key_version: "encryptionkey@0"
I20260924 13:44:46.358359 625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/wal/instance:
uuid: "c2a41ac0b004449b87463ed997467a04"
format_stamp: "Formatted at 2026-09-24 13:44:46 on dist-test-slave-tfm9"
server_key: "2936e99f247b06acfd8c2608917cdd1a"
server_key_iv: "778a546ec7ab0bdb6ccabe21df611e37"
server_key_version: "encryptionkey@0"
I20260924 13:44:46.359602 625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:44:46.360270 640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:46.360447 625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:44:46.360533 625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/wal
uuid: "c2a41ac0b004449b87463ed997467a04"
format_stamp: "Formatted at 2026-09-24 13:44:46 on dist-test-slave-tfm9"
server_key: "2936e99f247b06acfd8c2608917cdd1a"
server_key_iv: "778a546ec7ab0bdb6ccabe21df611e37"
server_key_version: "encryptionkey@0"
I20260924 13:44:46.360608 625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:46.368624 618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } attempt: 2
I20260924 13:44:46.369052 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.368902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:35141 (local address 127.0.0.1:57468)
0924 13:44:46.368957 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.369024 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 13:44:46.369473 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.369265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:35141 (local address 127.0.0.1:57470)
0924 13:44:46.369330 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.369450 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 13:44:46.369601 618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.253:35141: Network error: Client connection negotiation failed: client connection to 127.19.53.253:35141: connect: Connection refused (error 111)
I20260924 13:44:46.380694 625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:46.383100 625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:46.383271 625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:46.387547 625 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.253:35141
I20260924 13:44:46.387619 693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.253:35141 every 8 connection(s)
I20260924 13:44:46.387987 625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/data/info.pb
I20260924 13:44:46.388926 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:46.390792 694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.253@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:46.396888 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 625
I20260924 13:44:46.396991 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-1/wal/instance
I20260924 13:44:46.397176 19671 external_mini_cluster.cc:1428] Setting key 031cc3b50e512c86d7a60c22bb56f730
I20260924 13:44:46.399669 622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.391526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.19.53.254:43947)
0924 13:44:46.391582 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.391584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.391762 (+ 178us) server_negotiation.cc:407] Connection header received
0924 13:44:46.392189 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.392190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.392200 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.392269 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.392873 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.393412 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.394444 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.394658 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.394728 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.396805 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.396806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.396807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.396813 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.398301 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.398709 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.398711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.398711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.398778 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.399035 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.399035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.399036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.399149 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.399225 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.399497 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.399579 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:44:46.399861 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.391466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:43947 (local address 127.0.0.1:56868)
0924 13:44:46.391618 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.391656 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:46.392096 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:46.392353 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:46.392356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:46.392583 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:46.392774 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.392778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.393495 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.393496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.394371 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.394373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.394734 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.394734 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.394830 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.395306 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:46.395313 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:46.396685 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:46.398401 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.398408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.398410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.398632 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.398849 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.398850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.398850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.398960 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.399216 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:46.399222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:46.399367 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 13:44:46.399476 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 13:44:46.399625 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260924 13:44:46.400502 694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.53.252@KRBTEST.COM; defaulting to no policy
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.253@KRBTEST.COM for kudu/127.19.53.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.252@KRBTEST.COM" created.
I20260924 13:44:46.408501 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.400837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.19.53.253:35141)
0924 13:44:46.400972 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.400976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.400992 (+ 16us) server_negotiation.cc:407] Connection header received
0924 13:44:46.401012 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.401015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.401031 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.401086 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.401407 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.402143 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.403168 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.403442 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.403489 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.405391 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.405398 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.405398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.405404 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.407088 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.407530 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.407533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.407534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.407596 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.407913 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.407914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.407914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.408044 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.408107 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.408276 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.408401 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260924 13:44:46.408555 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.400780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:35141 (local address 127.0.0.1:57480)
0924 13:44:46.400839 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.400848 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:46.400880 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:46.401111 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:46.401112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:46.401264 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:46.401320 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.401321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.402236 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.402237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.403072 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.403074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.403514 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.403515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.403594 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.403938 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:46.403943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:46.405301 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:46.407214 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.407216 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.407216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.407430 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.407711 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.407712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.407713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.407835 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.408112 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:46.408113 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:46.408162 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 13:44:46.408242 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:44:46.408353 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 13:44:46.409466 694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:44:46.409814 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.409675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44717 (local address 127.0.0.1:37688)
0924 13:44:46.409718 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.409797 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 13:44:46.409956 644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
I20260924 13:44:46.410386 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.410297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44717 (local address 127.0.0.1:37692)
0924 13:44:46.410341 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.410364 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 13:44:46.410482 694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:44717: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.252.keytab.
Entry for principal kudu/127.19.53.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.252.keytab.
I20260924 13:44:46.421268 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.252:44717
--webserver_interface=127.19.53.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.252.keytab
--principal=kudu/127.19.53.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:43947,127.19.53.253:35141,127.19.53.252:44717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:44:46.459684 694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } attempt: 1
I20260924 13:44:46.460196 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.460063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44717 (local address 127.0.0.1:37708)
0924 13:44:46.460126 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.460165 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 13:44:46.460584 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.460456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44717 (local address 127.0.0.1:37720)
0924 13:44:46.460516 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.460557 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 13:44:46.460708 694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:44717: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
I20260924 13:44:46.460738 618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } attempt: 3
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for kudu/127.19.53.253@KRBTEST.COM
I20260924 13:44:46.467856 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.461056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.19.53.253:35141)
0924 13:44:46.461091 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.461093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.461110 (+ 17us) server_negotiation.cc:407] Connection header received
0924 13:44:46.461196 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.461197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.461207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.461266 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.461552 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.462347 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.462985 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.463304 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.463371 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.465475 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.465476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.465477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.465500 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.466836 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.467189 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.467190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.467190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.467237 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.467421 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.467421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.467422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.467535 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.467584 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.467735 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.467801 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:44:46.467898 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.461006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:35141 (local address 127.0.0.1:57488)
0924 13:44:46.461069 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.461079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:46.461125 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:46.461269 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:46.461270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:46.461419 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:46.461485 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.461486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.462433 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.462435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.462911 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.462912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.463385 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.463386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.463448 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.463964 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:46.463969 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:46.465403 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:46.466911 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.466912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.466913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.467115 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.467294 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.467294 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.467295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.467356 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.467593 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:46.467594 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:46.467640 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 13:44:46.467710 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:44:46.467806 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 13:44:46.468493 618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:44:46.468843 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.468738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44717 (local address 127.0.0.1:37730)
0924 13:44:46.468787 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.468819 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 13:44:46.469149 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.469070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44717 (local address 127.0.0.1:37736)
0924 13:44:46.469110 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.469134 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 13:44:46.469244 618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:44717: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
I20260924 13:44:46.500406 618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } attempt: 1
I20260924 13:44:46.500933 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.500743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44717 (local address 127.0.0.1:37740)
0924 13:44:46.500848 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.500895 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 13:44:46.501322 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.501216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44717 (local address 127.0.0.1:37746)
0924 13:44:46.501266 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.501298 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 13:44:46.501436 618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:44717: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
W20260924 13:44:46.503720 701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:46.503902 701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:46.503950 701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:46.505046 701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:44:46.505084 701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:46.505097 701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:46.505146 701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:44:46.505162 701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:44:46.506798 701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:43947,127.19.53.253:35141,127.19.53.252:44717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.252.keytab
--principal=kudu/127.19.53.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.252:44717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.53.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:46.507030 701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:46.507247 701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:46.512049 706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:46.512120 701 server_base.cc:1034] running on GCE node
W20260924 13:44:46.512058 709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:46.512058 707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:46.512506 701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:46.512835 701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:46.513988 701 hybrid_clock.cc:648] HybridClock initialized: now 1790257486513969 us; error 25 us; skew 500 ppm
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:46.516343 701 init.cc:377] Logged in from keytab as kudu/127.19.53.252@KRBTEST.COM (short username kudu)
I20260924 13:44:46.516767 701 webserver.cc:460] Webserver started at http://127.19.53.252:44755/ using document root <none> and password file <none>
I20260924 13:44:46.516940 701 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:46.516978 701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:46.517081 701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:44:46.518241 701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/data/instance:
uuid: "661a489cee2343a6b28e9557b0418451"
format_stamp: "Formatted at 2026-09-24 13:44:46 on dist-test-slave-tfm9"
server_key: "73e0a291a97730c20cc2ca2c5ae7b3f0"
server_key_iv: "939229138801b6c544a6457000e554fe"
server_key_version: "encryptionkey@0"
I20260924 13:44:46.518544 701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/wal/instance:
uuid: "661a489cee2343a6b28e9557b0418451"
format_stamp: "Formatted at 2026-09-24 13:44:46 on dist-test-slave-tfm9"
server_key: "73e0a291a97730c20cc2ca2c5ae7b3f0"
server_key_iv: "939229138801b6c544a6457000e554fe"
server_key_version: "encryptionkey@0"
I20260924 13:44:46.519776 701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:44:46.520408 716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:46.520596 701 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:44:46.520697 701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/wal
uuid: "661a489cee2343a6b28e9557b0418451"
format_stamp: "Formatted at 2026-09-24 13:44:46 on dist-test-slave-tfm9"
server_key: "73e0a291a97730c20cc2ca2c5ae7b3f0"
server_key_iv: "939229138801b6c544a6457000e554fe"
server_key_version: "encryptionkey@0"
I20260924 13:44:46.520787 701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:46.528853 694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } attempt: 2
I20260924 13:44:46.529227 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.529107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44717 (local address 127.0.0.1:37762)
0924 13:44:46.529163 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.529198 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 13:44:46.529575 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.529497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44717 (local address 127.0.0.1:37776)
0924 13:44:46.529539 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.529561 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 13:44:46.529677 694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:44717: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44717: connect: Connection refused (error 111)
I20260924 13:44:46.535949 701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:46.537747 701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:46.537863 701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:46.541471 701 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.252:44717
I20260924 13:44:46.541529 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.252:44717 every 8 connection(s)
I20260924 13:44:46.541858 701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/data/info.pb
I20260924 13:44:46.542809 769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:46.544307 769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:44:46.545975 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 701
I20260924 13:44:46.546077 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257335418480-19671-0/minicluster-data/master-2/wal/instance
I20260924 13:44:46.546281 19671 external_mini_cluster.cc:1428] Setting key 59ca88bb835d1ae826e8e00670cd99da
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.252@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.252@KRBTEST.COM
I20260924 13:44:46.554116 622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.544939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.19.53.254:43947)
0924 13:44:46.544998 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.545000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.545192 (+ 192us) server_negotiation.cc:407] Connection header received
0924 13:44:46.545688 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.545690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.545702 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.545752 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.546582 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.547338 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.548778 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.549022 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.549091 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.551312 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.551314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.551314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.551323 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.552698 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.553127 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.553128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.553129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.553189 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.553443 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.553444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.553444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.553567 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.553643 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.553936 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.554016 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:44:46.554329 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.544914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:43947 (local address 127.0.0.1:56872)
0924 13:44:46.545082 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.545116 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:46.545566 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:46.545825 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:46.545828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:46.546198 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:46.546468 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.546475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.547478 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.547478 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.548681 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.548683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.549098 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.549099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.549201 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.549709 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:46.549718 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:46.551218 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:46.552807 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.552814 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.552816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.553041 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.553274 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.553274 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.553275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.553376 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.553633 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:46.553639 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:46.553787 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 13:44:46.553897 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 13:44:46.554089 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":23,"thread_start_us":84,"threads_started":1}
I20260924 13:44:46.554955 769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:44:46.555850 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.546666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.19.53.252:44717)
0924 13:44:46.546830 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.546834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.546889 (+ 55us) server_negotiation.cc:407] Connection header received
0924 13:44:46.546958 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.546959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.546975 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.547036 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.547683 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.548817 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.549664 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.550100 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.550171 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.552626 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.552634 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.552634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.552640 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.554268 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.554709 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.554712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.554713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.554772 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.554974 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.554975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.554976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.555150 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.555309 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.555653 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.555733 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.252@KRBTEST.COM for kudu/127.19.53.253@KRBTEST.COM
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:46.561547 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.555221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:35141 (local address 127.0.0.1:57490)
0924 13:44:46.555274 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.555284 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:46.555318 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:46.555440 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:46.555440 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:46.555543 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:46.555592 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.555593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.556424 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.556425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.557095 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.557097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.557523 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.557524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.557606 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.557955 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:46.557960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:46.559055 (+ 1095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:46.560560 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.560561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.560561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.560705 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.560888 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.560888 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.560888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.560935 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.561247 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:46.561248 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:46.561304 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 13:44:46.561384 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:44:46.561473 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 13:44:46.561627 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.555295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.19.53.253:35141)
0924 13:44:46.555340 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.555342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.555370 (+ 28us) server_negotiation.cc:407] Connection header received
0924 13:44:46.555393 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.555393 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.555401 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.555438 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.555672 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.556357 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.557188 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.557453 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.557507 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.559144 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.559145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.559145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.559152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.560470 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.560771 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.560772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.560773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.560830 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.560992 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.560993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.560994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.561158 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.561233 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.561415 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.561517 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:44:46.561967 769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:44:46.564268 622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.556628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.19.53.254:43947)
0924 13:44:46.556662 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.556664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.556736 (+ 72us) server_negotiation.cc:407] Connection header received
0924 13:44:46.556763 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.556764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.556774 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.556859 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.557326 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.557949 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.558595 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.558810 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.558890 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.561254 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.561255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.561256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.561263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.563202 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.563531 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.563532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.563532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.563589 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.563768 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.563769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.563769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.563890 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.564071 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.564109 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.564178 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.252@KRBTEST.COM for kudu/127.19.53.252@KRBTEST.COM
I20260924 13:44:46.568667 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.562231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44717 (local address 127.0.0.1:37790)
0924 13:44:46.562278 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.562289 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:46.562325 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:46.562482 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:46.562483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:46.562633 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:46.562671 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.562672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.563303 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.563304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.564006 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.564008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.564491 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.564492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.564542 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.565007 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:46.565012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:46.566186 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:46.567716 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.567717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.567717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.567880 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.568077 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.568078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.568078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.568129 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.568369 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:46.568369 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:46.568404 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:44:46.568470 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 13:44:46.568559 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 13:44:46.568720 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.562289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.19.53.252:44717)
0924 13:44:46.562327 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.562329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.562337 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:44:46.562381 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.562382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.562390 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.562469 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.562756 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.563210 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.564104 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.564397 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.564481 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.566278 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.566279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.566279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.566285 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.567622 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.567962 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.567963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.567963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.568015 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.568198 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.568198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.568199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.568305 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.568377 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.568499 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.568605 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":20}
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.253@KRBTEST.COM
I20260924 13:44:46.570611 769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451: Bootstrap starting.
I20260924 13:44:46.571750 769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:46.571918 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.564912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.19.53.253:35141)
0924 13:44:46.564949 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.564951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.564959 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:44:46.564980 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.564981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.564991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.565044 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.565502 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.565980 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.566853 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.567162 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.567230 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.569390 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.569391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.569392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.569398 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.570770 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.571166 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.571166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.571167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.571212 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.571387 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.571388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.571388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.571477 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.571625 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.571715 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.571791 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:44:46.572194 769 log.cc:824] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:46.573104 769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451: No bootstrap required, opened a new log
I20260924 13:44:46.573673 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:44:46.574903 769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } }
I20260924 13:44:46.575058 769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:46.575098 769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661a489cee2343a6b28e9557b0418451, State: Initialized, Role: FOLLOWER
I20260924 13:44:46.575220 769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } }
I20260924 13:44:46.575538 769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:44:46.575484 782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } } }
I20260924 13:44:46.575610 782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:44:46.576592 618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } attempt: 2
I20260924 13:44:46.577611 701 master_runner.cc:186] Error getting master registration for 127.19.53.254:43947: OK, instance_id { permanent_uuid: "1c3612329f154f40a59f74f712d70603" instance_seqno: 1790257486234174 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:44:46.580139 701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:44:46.582831 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.575195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.19.53.253:35141)
0924 13:44:46.575452 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.575454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.575527 (+ 73us) server_negotiation.cc:407] Connection header received
0924 13:44:46.575617 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.575618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.575628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.575657 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.576362 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.577246 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.577973 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.578248 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.578323 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.579888 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.579889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.579890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.579897 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.581492 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.581857 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.581858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.581858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.581939 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.582127 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.582128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.582128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.582251 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.582372 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.582577 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.582650 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260924 13:44:46.584064 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.575291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.19.53.252:44717)
0924 13:44:46.575328 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.575330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.576444 (+ 1114us) server_negotiation.cc:407] Connection header received
0924 13:44:46.576526 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.576527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.576536 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.576563 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.577261 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.577944 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.578822 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.579101 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.579147 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.580577 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.580578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.580579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.580585 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.582744 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.583119 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.583120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.583120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.583181 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.583392 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.583393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.583393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.583574 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.583678 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.583879 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.583954 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 13:44:46.584309 793 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:44:46.584358 793 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for kudu/127.19.53.252@KRBTEST.COM
I20260924 13:44:46.588425 622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.575241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.19.53.254:43947)
0924 13:44:46.575487 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.575488 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.575496 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:44:46.575517 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.575518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.575528 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.575660 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.576514 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.580739 (+ 4225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.581755 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.582094 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.582135 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.584053 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.584055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.584055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.584061 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.586283 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.586626 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.586627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.586627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.586686 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.587681 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.587682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.587682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.587836 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.587924 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.588243 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.588331 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260924 13:44:46.588686 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.576826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44717 (local address 127.0.0.1:37796)
0924 13:44:46.576905 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.576916 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:46.576952 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:46.581012 (+ 4060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:46.581013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:46.581177 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:46.581219 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.581221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.582288 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.582289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.583074 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.583076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.583561 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.583562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.583621 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.584034 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:46.584040 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:46.585665 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:46.587618 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.587618 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.587619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.587837 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.588052 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.588052 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.588053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.588100 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.588365 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:46.588366 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:46.588406 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 13:44:46.588491 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 13:44:46.588581 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 13:44:46.588716 794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.577036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.19.53.252:44717)
0924 13:44:46.580875 (+ 3839us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.580877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.580884 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:44:46.580902 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.580902 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.580911 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.581311 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.581353 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.582174 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.583161 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.583486 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.583562 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.585777 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.585778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.585778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.585785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.587537 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.587916 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.587916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.587917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.587974 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.588175 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.588177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.588177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.588290 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.588359 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.588514 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.588586 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3817,"thread_start_us":60,"threads_started":1}
I20260924 13:44:46.590353 618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603: Bootstrap starting.
I20260924 13:44:46.590948 618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:46.591240 618 log.cc:824] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:46.591945 618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603: No bootstrap required, opened a new log
I20260924 13:44:46.593308 618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } }
I20260924 13:44:46.593411 618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:46.593432 618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3612329f154f40a59f74f712d70603, State: Initialized, Role: FOLLOWER
I20260924 13:44:46.593592 618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } }
I20260924 13:44:46.593843 797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } } }
I20260924 13:44:46.593936 618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:44:46.593945 797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 13:44:46.595933 808 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:44:46.595985 808 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:44:46.596618 549 master_runner.cc:186] Error getting master registration for 127.19.53.253:35141: OK, instance_id { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" instance_seqno: 1790257486386272 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:44:46.599295 549 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:44:46.620817 694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } attempt: 3
Sep 24 13:44:46 dist-test-slave-tfm9 krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257486, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.253@KRBTEST.COM for kudu/127.19.53.252@KRBTEST.COM
I20260924 13:44:46.629259 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.621072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44717 (local address 127.0.0.1:37806)
0924 13:44:46.621156 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:46.621184 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:46.621221 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:46.621535 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:46.621536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:46.621733 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:46.621784 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.621787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.622780 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.622781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.623607 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:46.623609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.624089 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.624090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:46.624147 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.624680 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:46.624685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:46.626236 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:46.628139 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.628140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.628141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.628368 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.628585 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:46.628586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:46.628586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:46.628654 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:46.628939 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:46.628940 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:46.628986 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 13:44:46.629057 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 13:44:46.629146 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 13:44:46.629264 794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:46.621335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.19.53.252:44717)
0924 13:44:46.621394 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:46.621396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:46.621404 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:44:46.621424 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:46.621424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:46.621435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:46.621486 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:46.621874 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.622697 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.623690 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:46.624010 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:46.624081 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:46.626335 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:46.626337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:46.626337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:46.626344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:46.628048 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.628447 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.628448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.628448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.628511 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:46.628724 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:46.628725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:46.628725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:46.628869 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:46.628936 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:46.629093 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 13:44:46.629173 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 13:44:46.630877 797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:44:46.630901 694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04: Bootstrap starting.
I20260924 13:44:46.630988 797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } }
I20260924 13:44:46.631232 797 leader_election.cc:288] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a41ac0b004449b87463ed997467a04 (127.19.53.253:35141), 661a489cee2343a6b28e9557b0418451 (127.19.53.252:44717)
I20260924 13:44:46.631655 694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:46.631732 743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c3612329f154f40a59f74f712d70603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661a489cee2343a6b28e9557b0418451" is_pre_election: true
I20260924 13:44:46.631888 743 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3612329f154f40a59f74f712d70603 in term 0.
I20260924 13:44:46.631959 694 log.cc:824] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:46.632108 568 leader_election.cc:302] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3612329f154f40a59f74f712d70603, 661a489cee2343a6b28e9557b0418451; no voters:
I20260924 13:44:46.632227 797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:44:46.632263 797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:44:46.632296 797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:46.632753 694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04: No bootstrap required, opened a new log
I20260924 13:44:46.633159 797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } }
I20260924 13:44:46.633294 797 leader_election.cc:288] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [CANDIDATE]: Term 1 election: Requested vote from peers c2a41ac0b004449b87463ed997467a04 (127.19.53.253:35141), 661a489cee2343a6b28e9557b0418451 (127.19.53.252:44717)
I20260924 13:44:46.633477 743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c3612329f154f40a59f74f712d70603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661a489cee2343a6b28e9557b0418451"
I20260924 13:44:46.633534 743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:46.634413 743 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3612329f154f40a59f74f712d70603 in term 1.
I20260924 13:44:46.634558 568 leader_election.cc:302] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3612329f154f40a59f74f712d70603, 661a489cee2343a6b28e9557b0418451; no voters:
I20260924 13:44:46.634814 694 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } }
I20260924 13:44:46.634958 694 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:46.634986 694 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a41ac0b004449b87463ed997467a04, State: Initialized, Role: FOLLOWER
I20260924 13:44:46.635092 694 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } }
I20260924 13:44:46.635372 694 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:44:46.635864 797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:44:46.636010 797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [term 1 LEADER]: Becoming Leader. State: Replica: 1c3612329f154f40a59f74f712d70603, State: Running, Role: LEADER
I20260924 13:44:46.636089 797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } }
I20260924 13:44:46.636425 797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3612329f154f40a59f74f712d70603. Latest consensus state: current_term: 1 leader_uuid: "1c3612329f154f40a59f74f712d70603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } } }
I20260924 13:44:46.636487 797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:46.636837 826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:44:46.636938 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c3612329f154f40a59f74f712d70603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a41ac0b004449b87463ed997467a04" is_pre_election: true
W20260924 13:44:46.636687 646 tablet.cc:2369] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:44:46.637054 667 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3612329f154f40a59f74f712d70603 in term 0.
I20260924 13:44:46.637148 666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c3612329f154f40a59f74f712d70603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a41ac0b004449b87463ed997467a04"
I20260924 13:44:46.637198 666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:46.637650 826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:44:46.637990 666 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3612329f154f40a59f74f712d70603 in term 1.
W20260924 13:44:46.639925 825 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:44:46.640021 825 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:44:46.640996 743 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [term 1 FOLLOWER]: Refusing update from remote peer 1c3612329f154f40a59f74f712d70603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:44:46.641197 666 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [term 1 FOLLOWER]: Refusing update from remote peer 1c3612329f154f40a59f74f712d70603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:44:46.642624 797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:44:46.642781 797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:44:46.644966 782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3612329f154f40a59f74f712d70603. Latest consensus state: current_term: 1 leader_uuid: "1c3612329f154f40a59f74f712d70603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } } }
I20260924 13:44:46.645042 782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:44:46.647704 797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c3612329f154f40a59f74f712d70603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } } }
I20260924 13:44:46.647780 797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:46.648238 782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c3612329f154f40a59f74f712d70603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } } }
I20260924 13:44:46.648327 782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:44:46.648576 827 mvcc.cc:205] Tried to move back new op lower bound from 7332894665278857216 to 7332894665261834240. Current Snapshot: MvccSnapshot[applied={T|T < 7332894665278857216}]
I20260924 13:44:46.648707 832 mvcc.cc:205] Tried to move back new op lower bound from 7332894665278857216 to 7332894665261834240. Current Snapshot: MvccSnapshot[applied={T|T < 7332894665278857216}]
I20260924 13:44:46.649158 797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c3612329f154f40a59f74f712d70603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } } }
I20260924 13:44:46.649226 797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603 [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:46.650007 811 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c3612329f154f40a59f74f712d70603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } } }
I20260924 13:44:46.650120 811 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:44:46.650179 811 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3612329f154f40a59f74f712d70603. Latest consensus state: current_term: 1 leader_uuid: "1c3612329f154f40a59f74f712d70603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } } }
I20260924 13:44:46.650247 811 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:44:46.650286 811 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c3612329f154f40a59f74f712d70603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3612329f154f40a59f74f712d70603" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43947 } } peers { permanent_uuid: "c2a41ac0b004449b87463ed997467a04" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 35141 } } peers { permanent_uuid: "661a489cee2343a6b28e9557b0418451" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44717 } } }
I20260924 13:44:46.650322 811 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:44:46.650429 830 mvcc.cc:205] Tried to move back new op lower bound from 7332894665278857216 to 7332894665261834240. Current Snapshot: MvccSnapshot[applied={T|T < 7332894665278857216}]
I20260924 13:44:46.654055 826 catalog_manager.cc:1347] Generated new cluster ID: 1f4fcc4e7c724a3fa3cc13596c7689d6
I20260924 13:44:46.654155 826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:44:46.675489 826 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:44:46.676918 826 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:44:46.684497 826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c3612329f154f40a59f74f712d70603: Generated new TSK 0
I20260924 13:44:46.684804 826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:44:47.584863 793 catalog_manager.cc:1259] Loaded cluster ID: 1f4fcc4e7c724a3fa3cc13596c7689d6
I20260924 13:44:47.584944 793 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451: loading cluster ID for follower catalog manager: success
I20260924 13:44:47.585819 793 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451: acquiring CA information for follower catalog manager: success
I20260924 13:44:47.586160 793 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 661a489cee2343a6b28e9557b0418451: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:44:47.640473 825 catalog_manager.cc:1259] Loaded cluster ID: 1f4fcc4e7c724a3fa3cc13596c7689d6
I20260924 13:44:47.640551 825 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04: loading cluster ID for follower catalog manager: success
I20260924 13:44:47.641350 825 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04: acquiring CA information for follower catalog manager: success
I20260924 13:44:47.641631 825 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2a41ac0b004449b87463ed997467a04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:44:47.748797 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 549
I20260924 13:44:47.755401 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 625
I20260924 13:44:47.761250 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 701
2026-09-24T13:44:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2792 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:44:47 dist-test-slave-tfm9 krb5kdc[846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:44:47 dist-test-slave-tfm9 krb5kdc[846](info): set up 2 sockets
Sep 24 13:44:47 dist-test-slave-tfm9 krb5kdc[846](info): commencing operation
krb5kdc: starting...
W20260924 13:44:48.815718 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:44:48 dist-test-slave-tfm9 krb5kdc[846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:44:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:44:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:44:48.928789 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:41263
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:37061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:41263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:44:49.007138 859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:49.007364 859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:49.007411 859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:49.008551 859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:44:49.008620 859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:49.008643 859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:49.008664 859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:44:49.008695 859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:44:49.010174 859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:37061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:41263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:41263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:49.010438 859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:49.010660 859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:49.015275 865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:49.015267 867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:49.015305 864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:49.015380 859 server_base.cc:1034] running on GCE node
I20260924 13:44:49.015785 859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:49.016096 859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:49.017266 859 hybrid_clock.cc:648] HybridClock initialized: now 1790257489017242 us; error 37 us; skew 500 ppm
Sep 24 13:44:49 dist-test-slave-tfm9 krb5kdc[846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257489, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:49.026779 859 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:44:49.027326 859 webserver.cc:460] Webserver started at http://127.19.53.254:45171/ using document root <none> and password file <none>
I20260924 13:44:49.027593 859 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:49.027647 859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:49.027834 859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:44:49.029030 859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "1942b04f265a4180a05d5030806a4028"
format_stamp: "Formatted at 2026-09-24 13:44:49 on dist-test-slave-tfm9"
server_key: "9ab882a22a56a531fc40843540ed71b9"
server_key_iv: "7be32d353e8e523a516545f55627acbd"
server_key_version: "encryptionkey@0"
I20260924 13:44:49.029412 859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "1942b04f265a4180a05d5030806a4028"
format_stamp: "Formatted at 2026-09-24 13:44:49 on dist-test-slave-tfm9"
server_key: "9ab882a22a56a531fc40843540ed71b9"
server_key_iv: "7be32d353e8e523a516545f55627acbd"
server_key_version: "encryptionkey@0"
I20260924 13:44:49.030992 859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:44:49.031821 874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:49.032001 859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:49.032119 859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "1942b04f265a4180a05d5030806a4028"
format_stamp: "Formatted at 2026-09-24 13:44:49 on dist-test-slave-tfm9"
server_key: "9ab882a22a56a531fc40843540ed71b9"
server_key_iv: "7be32d353e8e523a516545f55627acbd"
server_key_version: "encryptionkey@0"
I20260924 13:44:49.032217 859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:49.047755 859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:49.049738 859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:49.049952 859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:49.054052 859 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:41263
I20260924 13:44:49.054086 926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:41263 every 8 connection(s)
I20260924 13:44:49.054430 859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:44:49.055253 927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:49.057476 927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028: Bootstrap starting.
I20260924 13:44:49.058121 927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:49.058423 927 log.cc:824] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:49.059101 927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028: No bootstrap required, opened a new log
I20260924 13:44:49.060684 927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1942b04f265a4180a05d5030806a4028" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41263 } }
I20260924 13:44:49.060842 927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:49.060868 927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1942b04f265a4180a05d5030806a4028, State: Initialized, Role: FOLLOWER
I20260924 13:44:49.061036 927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1942b04f265a4180a05d5030806a4028" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41263 } }
I20260924 13:44:49.061100 927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:44:49.061122 927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:44:49.061149 927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:49.061676 927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1942b04f265a4180a05d5030806a4028" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41263 } }
I20260924 13:44:49.061760 927 leader_election.cc:302] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1942b04f265a4180a05d5030806a4028; no voters:
I20260924 13:44:49.061986 927 leader_election.cc:288] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:44:49.062026 930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:44:49.062177 930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [term 1 LEADER]: Becoming Leader. State: Replica: 1942b04f265a4180a05d5030806a4028, State: Running, Role: LEADER
I20260924 13:44:49.062249 927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:44:49.062343 930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1942b04f265a4180a05d5030806a4028" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41263 } }
I20260924 13:44:49.062907 932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1942b04f265a4180a05d5030806a4028. Latest consensus state: current_term: 1 leader_uuid: "1942b04f265a4180a05d5030806a4028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1942b04f265a4180a05d5030806a4028" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41263 } } }
I20260924 13:44:49.062999 932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:49.063161 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 859
I20260924 13:44:49.063252 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:44:49.063355 939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:44:49.063437 19671 external_mini_cluster.cc:1428] Setting key b092a888007c8f1bd66aae1f6ac75b93
I20260924 13:44:49.063385 931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1942b04f265a4180a05d5030806a4028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1942b04f265a4180a05d5030806a4028" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41263 } } }
I20260924 13:44:49.063478 931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028 [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:49.063962 939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:44:49.065559 939 catalog_manager.cc:1347] Generated new cluster ID: e3e42fe34cfd44dfb1c643b8d9200845
I20260924 13:44:49.065610 939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:44:49 dist-test-slave-tfm9 krb5kdc[846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:49.074074 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:49.063975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.19.53.254:41263)
0924 13:44:49.064744 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:49.064755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:49.064792 (+ 37us) server_negotiation.cc:407] Connection header received
0924 13:44:49.064901 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:49.064908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:49.064976 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:49.065121 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:49.065842 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:49.066870 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:49.067655 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:49.068098 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:49.068232 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:49.070815 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:49.070819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:49.070819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:49.070838 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:49.072469 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:49.072841 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:49.072844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:49.072845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:49.072953 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:49.073176 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:49.073177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:49.073177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:49.073410 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:49.073587 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:49.073701 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 13:44:49.073841 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":61,"threads_started":1}
I20260924 13:44:49.075037 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:44:49.090643 939 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:44:49.091418 939 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:44:49.100222 939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1942b04f265a4180a05d5030806a4028: Generated new TSK 0
I20260924 13:44:49.100417 939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:44:49.128794 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 859
2026-09-24T13:44:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1361 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:44:49 dist-test-slave-tfm9 krb5kdc[955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:44:49 dist-test-slave-tfm9 krb5kdc[955](info): set up 2 sockets
Sep 24 13:44:49 dist-test-slave-tfm9 krb5kdc[955](info): commencing operation
krb5kdc: starting...
W20260924 13:44:50.147718 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:44:50 dist-test-slave-tfm9 krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:44:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:44:50.264798 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33747
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:33747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:44:50.345147 968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:50.345422 968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:50.345445 968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:50.346642 968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:44:50.346680 968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:50.346694 968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:50.346707 968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:44:50.346719 968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:44:50.348131 968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:33747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:50.348320 968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:50.348515 968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:50.353269 974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:50.353327 976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:50.353241 973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:50.353408 968 server_base.cc:1034] running on GCE node
I20260924 13:44:50.353835 968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:50.354139 968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:50.355305 968 hybrid_clock.cc:648] HybridClock initialized: now 1790257490355292 us; error 43 us; skew 500 ppm
Sep 24 13:44:50 dist-test-slave-tfm9 krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257490, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:50.357668 968 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:44:50.358158 968 webserver.cc:460] Webserver started at http://127.19.53.254:45217/ using document root <none> and password file <none>
I20260924 13:44:50.358402 968 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:50.358467 968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:50.358613 968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:44:50.359637 968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "b14fe8c6eedf22e037258a9644f91dfd"
server_key_iv: "e299cce4fb5251a781889789bd34d0a7"
server_key_version: "encryptionkey@0"
I20260924 13:44:50.359984 968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "b14fe8c6eedf22e037258a9644f91dfd"
server_key_iv: "e299cce4fb5251a781889789bd34d0a7"
server_key_version: "encryptionkey@0"
I20260924 13:44:50.361326 968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:44:50.362149 983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:50.362327 968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:50.362437 968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "b14fe8c6eedf22e037258a9644f91dfd"
server_key_iv: "e299cce4fb5251a781889789bd34d0a7"
server_key_version: "encryptionkey@0"
I20260924 13:44:50.362524 968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:50.373275 968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:50.375015 968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:50.375171 968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:50.378866 968 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:33747
I20260924 13:44:50.378960 1036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:33747 every 8 connection(s)
I20260924 13:44:50.379279 968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:44:50.380163 1037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:50.382349 1037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Bootstrap starting.
I20260924 13:44:50.382895 1037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:50.383234 1037 log.cc:824] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:50.383905 1037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: No bootstrap required, opened a new log
I20260924 13:44:50.385438 1037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } }
I20260924 13:44:50.385564 1037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:50.385587 1037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e420cb8232a4df7b40f7ff2cfe2b51f, State: Initialized, Role: FOLLOWER
I20260924 13:44:50.385775 1037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } }
I20260924 13:44:50.385841 1037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:44:50.385862 1037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:44:50.385910 1037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:50.386791 1037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } }
I20260924 13:44:50.386898 1037 leader_election.cc:302] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e420cb8232a4df7b40f7ff2cfe2b51f; no voters:
I20260924 13:44:50.387065 1037 leader_election.cc:288] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:44:50.387116 1040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:44:50.387300 1040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 1 LEADER]: Becoming Leader. State: Replica: 8e420cb8232a4df7b40f7ff2cfe2b51f, State: Running, Role: LEADER
I20260924 13:44:50.387341 1037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:44:50.387487 1040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } }
I20260924 13:44:50.387848 1042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e420cb8232a4df7b40f7ff2cfe2b51f. Latest consensus state: current_term: 1 leader_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } } }
I20260924 13:44:50.387943 1042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:50.387837 1041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } } }
I20260924 13:44:50.388127 1041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:50.388386 1048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:44:50.388882 1048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:44:50.389178 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 968
I20260924 13:44:50.389318 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:44:50.389483 19671 external_mini_cluster.cc:1428] Setting key 9b65c2ecc4f508ca1d0fa0bc6ed337d7
I20260924 13:44:50.390595 1048 catalog_manager.cc:1347] Generated new cluster ID: 75729e012f9b4279b248f14258d7cf02
I20260924 13:44:50.390648 1048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:44:50 dist-test-slave-tfm9 krb5kdc[955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:50.399123 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.390179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.19.53.254:33747)
0924 13:44:50.390327 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:50.390336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:50.390369 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:44:50.390487 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:50.390512 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:50.390569 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:50.391327 (+ 758us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:50.391396 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.392250 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.392989 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.393380 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.393495 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.395738 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:50.395741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:50.395742 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:50.395753 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:50.397431 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:50.397821 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:50.397824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:50.397824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:50.397973 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:50.398189 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:50.398190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:50.398191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:50.398434 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:50.398608 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:50.398707 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 13:44:50.398948 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":72,"threads_started":1}
I20260924 13:44:50.405361 1048 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:44:50.406077 1048 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:44:50.415503 1048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Generated new TSK 0
I20260924 13:44:50.415678 1048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:44:50.449568 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33747
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:44:50.534389 1063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:50.534580 1063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:50.534603 1063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:50.535641 1063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:50.535682 1063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:50.535708 1063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:44:50.537070 1063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:50.537236 1063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:50.537400 1063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:50.542322 1069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:50.542433 1068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:50.542517 1071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:50.542666 1063 server_base.cc:1034] running on GCE node
I20260924 13:44:50.542900 1063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:50.543141 1063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:50.544329 1063 hybrid_clock.cc:648] HybridClock initialized: now 1790257490544314 us; error 28 us; skew 500 ppm
Sep 24 13:44:50 dist-test-slave-tfm9 krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257490, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:50.546967 1063 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:44:50.547417 1063 webserver.cc:460] Webserver started at http://127.19.53.193:41545/ using document root <none> and password file <none>
I20260924 13:44:50.547654 1063 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:50.547722 1063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:50.547876 1063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:44:50.548918 1063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "1401dffdd88a4bc48db013e9bc33a15f"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "03687ec81db6c5743550731ace883b03"
server_key_iv: "25778ed2ec4a1d25e7ed709998600537"
server_key_version: "encryptionkey@0"
I20260924 13:44:50.549283 1063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "1401dffdd88a4bc48db013e9bc33a15f"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "03687ec81db6c5743550731ace883b03"
server_key_iv: "25778ed2ec4a1d25e7ed709998600537"
server_key_version: "encryptionkey@0"
I20260924 13:44:50.550732 1063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:44:50.551465 1078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:50.551632 1063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:50.551728 1063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "1401dffdd88a4bc48db013e9bc33a15f"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "03687ec81db6c5743550731ace883b03"
server_key_iv: "25778ed2ec4a1d25e7ed709998600537"
server_key_version: "encryptionkey@0"
I20260924 13:44:50.551810 1063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:50.603824 1063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:50.606045 1063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:50.606209 1063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:50.606468 1063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:50.606798 1063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:44:50.606853 1063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:50.606889 1063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:44:50.606935 1063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:50.612497 1063 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:39533
I20260924 13:44:50.612594 1193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:39533 every 8 connection(s)
I20260924 13:44:50.612897 1063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:44:50.615252 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 1063
I20260924 13:44:50.615345 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:44:50.615556 19671 external_mini_cluster.cc:1428] Setting key 294254e2379cef5e1f7a5930e4a21129
Sep 24 13:44:50 dist-test-slave-tfm9 krb5kdc[955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257490, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:50.623493 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.613652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:48979 (local address 127.19.53.254:33747)
0924 13:44:50.613715 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:50.613717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:50.613897 (+ 180us) server_negotiation.cc:407] Connection header received
0924 13:44:50.614427 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:50.614428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:50.614439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:50.614491 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:50.615479 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.616175 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.617160 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.617494 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.617567 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.620242 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:50.620244 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:50.620245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:50.620252 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:50.622124 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:50.622472 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:50.622473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:50.622474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:50.622535 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:50.622816 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:50.622817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:50.622818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:50.622963 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:50.623039 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:50.623276 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 13:44:50.623376 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 13:44:50.623570 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.613605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:48979)
0924 13:44:50.613765 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:50.613806 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:50.614340 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:50.614552 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:50.614555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:50.615111 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:50.615343 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.615349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.616268 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.616269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.617066 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.617068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.617587 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.617588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.617747 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.618322 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:50.618335 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:50.620140 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:50.622207 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:50.622213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:50.622214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:50.622416 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:50.622630 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:50.622631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:50.622631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:50.622735 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:50.623033 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:50.623037 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:50.623145 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 13:44:50.623225 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:44:50.623355 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260924 13:44:50.624153 1194 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33747
I20260924 13:44:50.624260 1194 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:50.624433 1194 heartbeater.cc:511] Master 127.19.53.254:33747 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
I20260924 13:44:50.624984 1000 ts_manager.cc:194] Registered new tserver with Master: 1401dffdd88a4bc48db013e9bc33a15f (127.19.53.193:39533)
I20260924 13:44:50.625581 1000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:48979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:44:50.640654 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33747
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:44:50.721091 1199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:50.721282 1199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:50.721303 1199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:50.722373 1199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:50.722405 1199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:50.722430 1199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:44:50.723758 1199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:50.723927 1199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:50.724099 1199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:50.728940 1205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:50.728984 1204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:50.728940 1207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:50.729140 1199 server_base.cc:1034] running on GCE node
I20260924 13:44:50.729416 1199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:50.729652 1199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:50.730815 1199 hybrid_clock.cc:648] HybridClock initialized: now 1790257490730782 us; error 54 us; skew 500 ppm
Sep 24 13:44:50 dist-test-slave-tfm9 krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257490, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:50.733132 1199 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:44:50.733482 1199 webserver.cc:460] Webserver started at http://127.19.53.194:40473/ using document root <none> and password file <none>
I20260924 13:44:50.733705 1199 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:50.733749 1199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:50.733891 1199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:44:50.734954 1199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "70ea85a903954bd8ace09deb103299cc"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "25d4b5230c20a68ce9e19c4edb2348b3"
server_key_iv: "514181abc6ed50bc0b65e5ecc4aacfc8"
server_key_version: "encryptionkey@0"
I20260924 13:44:50.735296 1199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "70ea85a903954bd8ace09deb103299cc"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "25d4b5230c20a68ce9e19c4edb2348b3"
server_key_iv: "514181abc6ed50bc0b65e5ecc4aacfc8"
server_key_version: "encryptionkey@0"
I20260924 13:44:50.736645 1199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:44:50.737304 1214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:50.737478 1199 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:44:50.737563 1199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "70ea85a903954bd8ace09deb103299cc"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "25d4b5230c20a68ce9e19c4edb2348b3"
server_key_iv: "514181abc6ed50bc0b65e5ecc4aacfc8"
server_key_version: "encryptionkey@0"
I20260924 13:44:50.737632 1199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:50.752206 1199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:50.754031 1199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:50.754179 1199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:50.754412 1199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:50.754751 1199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:44:50.754789 1199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:50.754810 1199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:44:50.754822 1199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:50.760530 1199 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:43885
I20260924 13:44:50.760603 1329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:43885 every 8 connection(s)
I20260924 13:44:50.760931 1199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:44:50.765128 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 1199
I20260924 13:44:50.765259 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:44:50.765462 19671 external_mini_cluster.cc:1428] Setting key 0ffe9f09260a8ca6c3cbb664f1096299
Sep 24 13:44:50 dist-test-slave-tfm9 krb5kdc[955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257490, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:50.771145 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.761643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:58105 (local address 127.19.53.254:33747)
0924 13:44:50.761722 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:50.761748 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:50.761954 (+ 206us) server_negotiation.cc:407] Connection header received
0924 13:44:50.762495 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:50.762496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:50.762508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:50.762573 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:50.763676 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.764181 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.765205 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.765484 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.765565 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.768210 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:50.768211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:50.768212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:50.768218 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:50.769612 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:50.770100 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:50.770102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:50.770102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:50.770168 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:50.770486 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:50.770487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:50.770487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:50.770612 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:50.770688 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:50.770964 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 13:44:50.771053 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 13:44:50.771340 1332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.761617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:58105)
0924 13:44:50.761822 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:50.761858 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:50.762387 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:50.762653 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:50.762656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:50.763277 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:50.763571 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.763577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.764286 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.764286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.765115 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.765117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.765584 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.765585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.765747 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.766374 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:50.766385 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:50.768115 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:50.769717 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:50.769724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:50.769726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:50.770015 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:50.770233 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:50.770234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:50.770235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:50.770349 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:50.770679 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:50.770685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:50.770828 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 13:44:50.770932 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 13:44:50.771092 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260924 13:44:50.771873 1330 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33747
I20260924 13:44:50.771981 1330 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:50.772185 1330 heartbeater.cc:511] Master 127.19.53.254:33747 requested a full tablet report, sending...
I20260924 13:44:50.772557 1000 ts_manager.cc:194] Registered new tserver with Master: 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885)
I20260924 13:44:50.772948 1000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:58105
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:44:50.792095 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33747
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:44:50.873639 1335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:50.873848 1335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:50.873871 1335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:50.875026 1335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:50.875068 1335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:50.875095 1335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:44:50.876570 1335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:50.876753 1335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:50.876921 1335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:50.881727 1343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:50.881706 1341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:50.881765 1335 server_base.cc:1034] running on GCE node
W20260924 13:44:50.881706 1340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:50.882156 1335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:50.882436 1335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:50.883603 1335 hybrid_clock.cc:648] HybridClock initialized: now 1790257490883590 us; error 25 us; skew 500 ppm
Sep 24 13:44:50 dist-test-slave-tfm9 krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257490, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:50.885978 1335 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:44:50.886467 1335 webserver.cc:460] Webserver started at http://127.19.53.195:33393/ using document root <none> and password file <none>
I20260924 13:44:50.886693 1335 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:50.886757 1335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:50.886893 1335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:44:50.888026 1335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "d80aae5f591e4114b26c16912088d070"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "832900b590bf9d5c82798e09ccade414"
server_key_iv: "4e450991ba43341707fd6629cb695c52"
server_key_version: "encryptionkey@0"
I20260924 13:44:50.888434 1335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "d80aae5f591e4114b26c16912088d070"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "832900b590bf9d5c82798e09ccade414"
server_key_iv: "4e450991ba43341707fd6629cb695c52"
server_key_version: "encryptionkey@0"
I20260924 13:44:50.889848 1335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:44:50.890590 1350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:50.890784 1335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:44:50.890878 1335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "d80aae5f591e4114b26c16912088d070"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "832900b590bf9d5c82798e09ccade414"
server_key_iv: "4e450991ba43341707fd6629cb695c52"
server_key_version: "encryptionkey@0"
I20260924 13:44:50.890964 1335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:50.915025 1335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:50.917156 1335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:50.917320 1335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:50.917544 1335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:50.917842 1335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:44:50.917913 1335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:50.917979 1335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:44:50.918007 1335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:50.923866 1335 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:43169
I20260924 13:44:50.923902 1465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:43169 every 8 connection(s)
I20260924 13:44:50.924243 1335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:44:50.926942 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 1335
I20260924 13:44:50.927011 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:44:50.927179 19671 external_mini_cluster.cc:1428] Setting key a9032a9fba95b776a853a423e687ce3e
Sep 24 13:44:50 dist-test-slave-tfm9 krb5kdc[955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257490, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:50.933707 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.925025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:48691 (local address 127.19.53.254:33747)
0924 13:44:50.925094 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:50.925096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:50.925287 (+ 191us) server_negotiation.cc:407] Connection header received
0924 13:44:50.925758 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:50.925759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:50.925769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:50.925826 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:50.926783 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.927451 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.928699 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.928890 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.928947 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.930942 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:50.930944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:50.930944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:50.930951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:50.932524 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:50.932929 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:50.932930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:50.932931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:50.932987 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:50.933216 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:50.933217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:50.933217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:50.933318 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:50.933368 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:50.933587 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 13:44:50.933646 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 13:44:50.933928 1468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.924999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:48691)
0924 13:44:50.925157 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:50.925199 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:50.925672 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:50.925880 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:50.925883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:50.926507 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:50.926696 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.926701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.927593 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.927594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.928623 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.928626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.928963 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.928964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.929062 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.929463 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:50.929471 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:50.930833 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:50.932612 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:50.932617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:50.932619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:50.932846 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:50.933062 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:50.933062 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:50.933063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:50.933153 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:50.933372 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:50.933376 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:50.933480 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 13:44:50.933559 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 13:44:50.933666 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260924 13:44:50.934393 1466 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33747
I20260924 13:44:50.934470 1466 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:50.934648 1466 heartbeater.cc:511] Master 127.19.53.254:33747 requested a full tablet report, sending...
I20260924 13:44:50.934989 1000 ts_manager.cc:194] Registered new tserver with Master: d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169)
I20260924 13:44:50.935382 1000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:48691
I20260924 13:44:50.938868 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:44:50.945690 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.940156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49996 (local address 127.19.53.254:33747)
0924 13:44:50.940202 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:50.940204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:50.940342 (+ 138us) server_negotiation.cc:407] Connection header received
0924 13:44:50.940393 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:50.940394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:50.940402 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:50.940449 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:50.941001 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.941498 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.942242 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.942425 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.942489 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.943642 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:50.943643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:50.943643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:50.943647 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:50.944785 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:50.945074 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:50.945074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:50.945075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:50.945121 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:50.945273 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:50.945273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:50.945274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:50.945414 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:50.945482 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:50.945580 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 13:44:50.945635 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:44:50.947348 1000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:44:50.948889 1000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:44:50.955458 1478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.951215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:43885 (local address 127.0.0.1:36768)
0924 13:44:50.951435 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:50.951478 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:50.951551 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:50.952085 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:50.952086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:50.952088 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:50.952129 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.952130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.953330 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.953331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.954279 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.954281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.955205 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.955205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.955283 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.955297 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 13:44:50.955369 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 13:44:50.955406 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260924 13:44:50.955724 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.951328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.19.53.194:43885)
0924 13:44:50.951902 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:50.951907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:50.951917 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:44:50.951955 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:50.951957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:50.951975 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:50.952197 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:50.952269 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.953237 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.954363 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.955127 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.955203 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.955262 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:50.955402 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 13:44:50.955445 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":96,"threads_started":1}
I20260924 13:44:50.955845 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.951215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39533 (local address 127.0.0.1:39902)
0924 13:44:50.951519 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:50.951529 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:50.951579 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:50.951763 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:50.951767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:50.951770 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:50.951837 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.951839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.953138 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.953139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.954382 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.954386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.955608 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.955609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.955669 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.955675 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 13:44:50.955741 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 13:44:50.955783 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":6,"thread_start_us":47,"threads_started":1}
I20260924 13:44:50.955951 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.951422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39902 (local address 127.19.53.193:39533)
0924 13:44:50.951578 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:50.951583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:50.951595 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:44:50.951638 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:50.951640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:50.951657 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:50.951737 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:50.951934 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.953038 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.954457 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.955127 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.955203 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.955271 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:50.955800 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 13:44:50.955845 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260924 13:44:50.956511 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.951404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:43169 (local address 127.0.0.1:48222)
0924 13:44:50.951682 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:50.951691 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:50.951777 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:50.952146 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:50.952147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:50.952148 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:50.952197 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.952199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.952989 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.952990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.954394 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.954396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.956298 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.956299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.956358 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.956363 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:44:50.956410 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 13:44:50.956448 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":30,"thread_start_us":68,"threads_started":1}
I20260924 13:44:50.956727 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.951660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.19.53.195:43169)
0924 13:44:50.951962 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:50.951967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:50.951976 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:44:50.952013 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:50.952014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:50.952031 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:50.952147 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:50.952254 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.952895 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.955319 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.956126 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.956196 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.956252 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:50.956609 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 13:44:50.956656 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":133,"threads_started":1}
I20260924 13:44:50.956624 1128 tablet_service.cc:1467] Processing CreateTablet for tablet e4b9f47871f14e03b4c9e2df89fd2811 (DEFAULT_TABLE table=test-table [id=a24f5c71874e4624a9e4053748d23ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:44:50.956619 1262 tablet_service.cc:1467] Processing CreateTablet for tablet e4b9f47871f14e03b4c9e2df89fd2811 (DEFAULT_TABLE table=test-table [id=a24f5c71874e4624a9e4053748d23ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:44:50.956998 1128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b9f47871f14e03b4c9e2df89fd2811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:50.956977 1262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b9f47871f14e03b4c9e2df89fd2811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:50.957394 1399 tablet_service.cc:1467] Processing CreateTablet for tablet e4b9f47871f14e03b4c9e2df89fd2811 (DEFAULT_TABLE table=test-table [id=a24f5c71874e4624a9e4053748d23ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:44:50.957681 1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b9f47871f14e03b4c9e2df89fd2811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:50.959723 1483 tablet_bootstrap.cc:492] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Bootstrap starting.
I20260924 13:44:50.960341 1484 tablet_bootstrap.cc:492] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc: Bootstrap starting.
I20260924 13:44:50.960374 1483 tablet_bootstrap.cc:654] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:50.961026 1484 tablet_bootstrap.cc:654] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:50.961011 1483 log.cc:824] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:50.961119 1485 tablet_bootstrap.cc:492] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070: Bootstrap starting.
I20260924 13:44:50.961571 1484 log.cc:824] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:50.961768 1485 tablet_bootstrap.cc:654] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:50.961836 1483 tablet_bootstrap.cc:492] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: No bootstrap required, opened a new log
I20260924 13:44:50.961905 1483 ts_tablet_manager.cc:1397] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:44:50.962347 1484 tablet_bootstrap.cc:492] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc: No bootstrap required, opened a new log
I20260924 13:44:50.962353 1485 log.cc:824] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:50.962455 1484 ts_tablet_manager.cc:1397] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:44:50.962992 1485 tablet_bootstrap.cc:492] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070: No bootstrap required, opened a new log
I20260924 13:44:50.963078 1485 ts_tablet_manager.cc:1397] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:44:50.964169 1483 raft_consensus.cc:348] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:50.964344 1483 raft_consensus.cc:374] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:50.964382 1483 raft_consensus.cc:729] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1401dffdd88a4bc48db013e9bc33a15f, State: Initialized, Role: FOLLOWER
I20260924 13:44:50.964396 1484 raft_consensus.cc:348] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:50.964540 1484 raft_consensus.cc:374] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:50.964572 1484 raft_consensus.cc:729] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ea85a903954bd8ace09deb103299cc, State: Initialized, Role: FOLLOWER
I20260924 13:44:50.964525 1483 consensus_queue.cc:260] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:50.964452 1485 raft_consensus.cc:348] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:50.964644 1485 raft_consensus.cc:374] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:50.964684 1485 raft_consensus.cc:729] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80aae5f591e4114b26c16912088d070, State: Initialized, Role: FOLLOWER
I20260924 13:44:50.964711 1484 consensus_queue.cc:260] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:50.964836 1483 ts_tablet_manager.cc:1428] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:44:50.964826 1485 consensus_queue.cc:260] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:50.964994 1484 ts_tablet_manager.cc:1428] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:44:50.964982 1194 heartbeater.cc:503] Master 127.19.53.254:33747 was elected leader, sending a full tablet report...
I20260924 13:44:50.965138 1485 ts_tablet_manager.cc:1428] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:44:50.965153 1466 heartbeater.cc:503] Master 127.19.53.254:33747 was elected leader, sending a full tablet report...
I20260924 13:44:50.965291 1330 heartbeater.cc:503] Master 127.19.53.254:33747 was elected leader, sending a full tablet report...
I20260924 13:44:50.979009 1490 raft_consensus.cc:482] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:44:50.979089 1490 raft_consensus.cc:504] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:50.979457 1490 leader_election.cc:288] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169), 1401dffdd88a4bc48db013e9bc33a15f (127.19.53.193:39533)
I20260924 13:44:50.983283 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.979719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:50463 (local address 127.19.53.193:39533)
0924 13:44:50.979801 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:50.979803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:50.979811 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:44:50.979831 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:50.979832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:50.979839 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:50.979888 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:50.980077 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.980853 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.982335 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.982925 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.982998 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.983005 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:50.983177 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 13:44:50.983216 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 13:44:50.983292 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.979679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:44957 (local address 127.19.53.195:43169)
0924 13:44:50.979727 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:50.979729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:50.979924 (+ 195us) server_negotiation.cc:407] Connection header received
0924 13:44:50.979949 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:50.979950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:50.979958 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:50.980006 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:50.980147 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.980846 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.982336 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.982919 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.982997 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.983005 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:50.983177 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 13:44:50.983216 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 13:44:50.983306 1332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.979626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39533 (local address 127.19.53.194:50463)
0924 13:44:50.979714 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:50.979724 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:50.979774 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:50.979911 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:50.979912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:50.979914 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:50.980012 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.980014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.980927 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.980927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.982250 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.982253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.983015 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.983016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.983100 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.983101 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:44:50.983158 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 13:44:50.983208 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":91}
I20260924 13:44:50.983314 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.979627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:43169 (local address 127.19.53.194:44957)
0924 13:44:50.979851 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:50.979858 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:50.979911 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:50.980025 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:50.980026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:50.980027 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:50.980087 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.980088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.980927 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.980927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.982265 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:50.982267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.983015 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.983016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:50.983097 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.983101 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:44:50.983158 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 13:44:50.983208 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":114,"thread_start_us":87,"threads_started":1}
I20260924 13:44:50.983625 1148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b9f47871f14e03b4c9e2df89fd2811" candidate_uuid: "70ea85a903954bd8ace09deb103299cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1401dffdd88a4bc48db013e9bc33a15f" is_pre_election: true
I20260924 13:44:50.983779 1148 raft_consensus.cc:2454] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70ea85a903954bd8ace09deb103299cc in term 0.
I20260924 13:44:50.983915 1419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b9f47871f14e03b4c9e2df89fd2811" candidate_uuid: "70ea85a903954bd8ace09deb103299cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d80aae5f591e4114b26c16912088d070" is_pre_election: true
I20260924 13:44:50.984001 1216 leader_election.cc:302] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1401dffdd88a4bc48db013e9bc33a15f, 70ea85a903954bd8ace09deb103299cc; no voters:
I20260924 13:44:50.984042 1419 raft_consensus.cc:2454] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70ea85a903954bd8ace09deb103299cc in term 0.
I20260924 13:44:50.984223 1490 raft_consensus.cc:2781] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:44:50.984272 1490 raft_consensus.cc:482] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:44:50.984287 1490 raft_consensus.cc:3037] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:50.984905 1490 raft_consensus.cc:504] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:50.985056 1490 leader_election.cc:288] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [CANDIDATE]: Term 1 election: Requested vote from peers d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169), 1401dffdd88a4bc48db013e9bc33a15f (127.19.53.193:39533)
I20260924 13:44:50.985196 1419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b9f47871f14e03b4c9e2df89fd2811" candidate_uuid: "70ea85a903954bd8ace09deb103299cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d80aae5f591e4114b26c16912088d070"
I20260924 13:44:50.985206 1148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b9f47871f14e03b4c9e2df89fd2811" candidate_uuid: "70ea85a903954bd8ace09deb103299cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1401dffdd88a4bc48db013e9bc33a15f"
I20260924 13:44:50.985281 1419 raft_consensus.cc:3037] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:50.985281 1148 raft_consensus.cc:3037] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:50.986160 1419 raft_consensus.cc:2454] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70ea85a903954bd8ace09deb103299cc in term 1.
I20260924 13:44:50.986160 1148 raft_consensus.cc:2454] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70ea85a903954bd8ace09deb103299cc in term 1.
I20260924 13:44:50.986323 1216 leader_election.cc:302] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1401dffdd88a4bc48db013e9bc33a15f, 70ea85a903954bd8ace09deb103299cc; no voters:
I20260924 13:44:50.986419 1490 raft_consensus.cc:2781] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:44:50.986544 1490 raft_consensus.cc:686] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [term 1 LEADER]: Becoming Leader. State: Replica: 70ea85a903954bd8ace09deb103299cc, State: Running, Role: LEADER
I20260924 13:44:50.986635 1490 consensus_queue.cc:237] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:50.987375 999 catalog_manager.cc:5495] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 70ea85a903954bd8ace09deb103299cc (127.19.53.194). New cstate: current_term: 1 leader_uuid: "70ea85a903954bd8ace09deb103299cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } health_report { overall_health: UNKNOWN } } }
I20260924 13:44:50.999861 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:50.997779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.19.53.194:43885)
0924 13:44:50.997815 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:50.997817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:50.997834 (+ 17us) server_negotiation.cc:407] Connection header received
0924 13:44:50.997901 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:50.997902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:50.997958 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:50.998034 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:44:50.998158 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.998592 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.999271 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:50.999442 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:50.999490 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:50.999616 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:44:50.999700 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:44:50.999763 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:50.999782 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 13:44:50.999797 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:44:51.001504 1148 raft_consensus.cc:1261] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Refusing update from remote peer 70ea85a903954bd8ace09deb103299cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:44:51.001511 1419 raft_consensus.cc:1261] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070 [term 1 FOLLOWER]: Refusing update from remote peer 70ea85a903954bd8ace09deb103299cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:44:51.001904 1493 consensus_queue.cc:1035] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:44:51.002060 1490 consensus_queue.cc:1035] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:44:51.004309 1498 mvcc.cc:205] Tried to move back new op lower bound from 7332894683139411968 to 7332894683081830400. Current Snapshot: MvccSnapshot[applied={T|T < 7332894683139411968}]
I20260924 13:44:51.004637 1497 mvcc.cc:205] Tried to move back new op lower bound from 7332894683139411968 to 7332894683081830400. Current Snapshot: MvccSnapshot[applied={T|T < 7332894683139411968}]
I20260924 13:44:51.006963 998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:44:51.007050 998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:44:51.007846 998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Sending DeleteTablet for 3 replicas of tablet e4b9f47871f14e03b4c9e2df89fd2811
I20260924 13:44:51.008198 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 1063
I20260924 13:44:51.008177 1128 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b9f47871f14e03b4c9e2df89fd2811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:44:51 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:39902
I20260924 13:44:51.008193 1399 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b9f47871f14e03b4c9e2df89fd2811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:44:51 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:48222
I20260924 13:44:51.008678 1262 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b9f47871f14e03b4c9e2df89fd2811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:44:51 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:36768
I20260924 13:44:51.008737 1506 tablet_replica.cc:330] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070: stopping tablet replica
I20260924 13:44:51.008834 1506 raft_consensus.cc:2229] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:44:51.008904 1507 tablet_replica.cc:330] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc: stopping tablet replica
I20260924 13:44:51.008939 1506 raft_consensus.cc:2258] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:44:51.008994 1507 raft_consensus.cc:2229] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:44:51.009100 1507 raft_consensus.cc:2258] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:44:51.009455 1506 ts_tablet_manager.cc:1905] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:44:51.009470 1507 ts_tablet_manager.cc:1905] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:44:51.011394 1507 ts_tablet_manager.cc:1918] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:44:51.011397 1506 ts_tablet_manager.cc:1918] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:44:51.011466 1507 log.cc:1196] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/e4b9f47871f14e03b4c9e2df89fd2811
I20260924 13:44:51.011467 1506 log.cc:1196] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/e4b9f47871f14e03b4c9e2df89fd2811
I20260924 13:44:51.011746 1506 ts_tablet_manager.cc:1939] T e4b9f47871f14e03b4c9e2df89fd2811 P d80aae5f591e4114b26c16912088d070: Deleting consensus metadata
I20260924 13:44:51.011759 1507 ts_tablet_manager.cc:1939] T e4b9f47871f14e03b4c9e2df89fd2811 P 70ea85a903954bd8ace09deb103299cc: Deleting consensus metadata
I20260924 13:44:51.012156 987 catalog_manager.cc:4841] TS 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885): tablet e4b9f47871f14e03b4c9e2df89fd2811 (table test-table [id=a24f5c71874e4624a9e4053748d23ba3]) successfully deleted
W20260924 13:44:51.013931 985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:39533 (error 108)
I20260924 13:44:51.014238 987 catalog_manager.cc:4841] TS d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169): tablet e4b9f47871f14e03b4c9e2df89fd2811 (table test-table [id=a24f5c71874e4624a9e4053748d23ba3]) successfully deleted
W20260924 13:44:51.014454 1480 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:44:51.014308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39533 (local address 127.0.0.1:39910)
0924 13:44:51.014365 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:51.014399 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:39533: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 13:44:51.014564 985 catalog_manager.cc:4584] TS 1401dffdd88a4bc48db013e9bc33a15f (127.19.53.193:39533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4b9f47871f14e03b4c9e2df89fd2811: Network error: Client connection negotiation failed: client connection to 127.19.53.193:39533: connect: Connection reset by peer (error 104)
I20260924 13:44:51.014732 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 1199
I20260924 13:44:51.019131 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 1335
I20260924 13:44:51.023581 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 968
I20260924 13:44:51.028528 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33747
--webserver_interface=127.19.53.254
--webserver_port=45217
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:33747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:44:51.107892 1509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:51.108088 1509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:51.108110 1509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:51.109257 1509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:44:51.109299 1509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:51.109313 1509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:51.109373 1509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:44:51.109398 1509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:44:51.110898 1509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:33747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=45217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:51.111159 1509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:51.111392 1509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:51.119863 1515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:51.120095 1509 server_base.cc:1034] running on GCE node
W20260924 13:44:51.119927 1518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:51.119927 1516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:51.120481 1509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:51.120852 1509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:51.121977 1509 hybrid_clock.cc:648] HybridClock initialized: now 1790257491121969 us; error 37 us; skew 500 ppm
Sep 24 13:44:51 dist-test-slave-tfm9 krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257491, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:51.124565 1509 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:44:51.125038 1509 webserver.cc:460] Webserver started at http://127.19.53.254:45217/ using document root <none> and password file <none>
I20260924 13:44:51.125262 1509 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:51.125312 1509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:51.126730 1509 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:51.127379 1525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:51.127585 1509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:51.127669 1509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "b14fe8c6eedf22e037258a9644f91dfd"
server_key_iv: "e299cce4fb5251a781889789bd34d0a7"
server_key_version: "encryptionkey@0"
I20260924 13:44:51.128063 1509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:51.158512 1509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:51.160348 1509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:51.160458 1509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:51.164323 1509 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:33747
I20260924 13:44:51.164433 1577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:33747 every 8 connection(s)
I20260924 13:44:51.164799 1509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:44:51.166508 1578 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:44:51.167235 1578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Bootstrap starting.
I20260924 13:44:51.173530 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 1509
I20260924 13:44:51.173844 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:39533
--local_ip_for_outbound_sockets=127.19.53.193
--tserver_master_addrs=127.19.53.254:33747
--webserver_port=41545
--webserver_interface=127.19.53.193
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:44:51.175979 1578 log.cc:824] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:51.178063 1578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:44:51.178381 1578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Bootstrap complete.
I20260924 13:44:51.180660 1578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } }
I20260924 13:44:51.180905 1578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e420cb8232a4df7b40f7ff2cfe2b51f, State: Initialized, Role: FOLLOWER
I20260924 13:44:51.181094 1578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } }
I20260924 13:44:51.181185 1578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:44:51.181245 1578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:44:51.181319 1578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:44:51.182467 1578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } }
I20260924 13:44:51.182608 1578 leader_election.cc:302] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e420cb8232a4df7b40f7ff2cfe2b51f; no voters:
I20260924 13:44:51.182832 1578 leader_election.cc:288] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 13:44:51.182874 1582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:44:51.182994 1582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 2 LEADER]: Becoming Leader. State: Replica: 8e420cb8232a4df7b40f7ff2cfe2b51f, State: Running, Role: LEADER
I20260924 13:44:51.183084 1582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } }
I20260924 13:44:51.183095 1578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:44:51.184284 1583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } } }
I20260924 13:44:51.184399 1583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:51.184664 1584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e420cb8232a4df7b40f7ff2cfe2b51f. Latest consensus state: current_term: 2 leader_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } } }
I20260924 13:44:51.184762 1584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:51.185103 1599 catalog_manager.cc:1259] Loaded cluster ID: 75729e012f9b4279b248f14258d7cf02
I20260924 13:44:51.185148 1599 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: loading cluster ID for follower catalog manager: success
W20260924 13:44:51.185329 1599 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260924 13:44:51.185365 1599 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260924 13:44:51.185406 1598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:44:51.185747 1598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:44:51.185828 1598 catalog_manager.cc:1259] Loaded cluster ID: 75729e012f9b4279b248f14258d7cf02
I20260924 13:44:51.185855 1598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:44:51.186026 1598 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 13:44:51.191111 1598 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 13:44:51.193939 1598 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:44:51.194113 1598 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Loaded TSK: 0
I20260924 13:44:51.194345 1598 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:44:51.257303 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:51.257488 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:51.257511 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:51.258687 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:51.258726 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:51.258754 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:44:51.260167 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:39533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=41545
--tserver_master_addrs=127.19.53.254:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:51.260397 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:51.260637 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:51.265560 1607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:51.265590 1606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:51.265825 1580 server_base.cc:1034] running on GCE node
W20260924 13:44:51.265823 1609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:51.266148 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:51.266415 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:51.267570 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790257491267561 us; error 25 us; skew 500 ppm
Sep 24 13:44:51 dist-test-slave-tfm9 krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257491, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:51.270190 1580 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:44:51.270704 1580 webserver.cc:460] Webserver started at http://127.19.53.193:41545/ using document root <none> and password file <none>
I20260924 13:44:51.270936 1580 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:51.271018 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:51.272325 1580 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 13:44:51.273003 1616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:51.273345 1580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:51.273458 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "1401dffdd88a4bc48db013e9bc33a15f"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "03687ec81db6c5743550731ace883b03"
server_key_iv: "25778ed2ec4a1d25e7ed709998600537"
server_key_version: "encryptionkey@0"
I20260924 13:44:51.273931 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:51.291186 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:51.293583 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:51.293763 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:51.294059 1580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:51.294559 1623 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 13:44:51.295917 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 13:44:51.295979 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 13:44:51.296017 1580 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 13:44:51.296854 1580 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 13:44:51.296903 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 13:44:51.296962 1623 tablet_bootstrap.cc:492] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Bootstrap starting.
I20260924 13:44:51.304713 1580 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:39533
I20260924 13:44:51.304904 1731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:39533 every 8 connection(s)
I20260924 13:44:51.305179 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:44:51.307699 1623 log.cc:824] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:51.308918 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 1580
I20260924 13:44:51.309237 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:43885
--local_ip_for_outbound_sockets=127.19.53.194
--tserver_master_addrs=127.19.53.254:33747
--webserver_port=40473
--webserver_interface=127.19.53.194
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:44:51.310304 1623 tablet_bootstrap.cc:492] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:44:51.310614 1623 tablet_bootstrap.cc:492] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Bootstrap complete.
I20260924 13:44:51.311888 1623 ts_tablet_manager.cc:1397] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260924 13:44:51.313261 1623 raft_consensus.cc:348] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:51.313578 1623 raft_consensus.cc:729] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1401dffdd88a4bc48db013e9bc33a15f, State: Initialized, Role: FOLLOWER
Sep 24 13:44:51 dist-test-slave-tfm9 krb5kdc[955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257491, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:51.318029 1623 consensus_queue.cc:260] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:51.318413 1623 ts_tablet_manager.cc:1428] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260924 13:44:51.320370 1735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.305991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:53539 (local address 127.19.53.254:33747)
0924 13:44:51.306213 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:51.306224 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:51.306289 (+ 65us) server_negotiation.cc:407] Connection header received
0924 13:44:51.307062 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:51.307072 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:51.307143 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:51.307311 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:51.308269 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.309679 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.312202 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.312825 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.312951 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.315712 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:51.315717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:51.315717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:51.315730 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:51.318135 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:51.318754 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:51.318758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:51.318758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:51.318903 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:51.319250 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:51.319251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:51.319251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:51.319538 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:51.319632 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:51.320027 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 13:44:51.320170 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":64,"threads_started":1}
I20260924 13:44:51.320384 1734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.305872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:53539)
0924 13:44:51.306115 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:51.306163 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:51.306777 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:51.307357 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:51.307362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:51.307808 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:51.308088 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.308094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.310815 (+ 2721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.310816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.312110 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.312113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.312952 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.312953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.313110 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.313745 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:51.313756 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:51.315591 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:51.318240 (+ 2649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:51.318246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:51.318248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:51.318627 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:51.319000 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:51.319002 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:51.319002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:51.319153 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:51.319626 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:51.319633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:51.319803 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 13:44:51.319925 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 13:44:51.320113 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":133,"threads_started":1}
I20260924 13:44:51.321586 1732 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33747
I20260924 13:44:51.321687 1732 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:51.321875 1732 heartbeater.cc:511] Master 127.19.53.254:33747 requested a full tablet report, sending...
I20260924 13:44:51.322731 1542 ts_manager.cc:194] Registered new tserver with Master: 1401dffdd88a4bc48db013e9bc33a15f (127.19.53.193:39533)
I20260924 13:44:51.323958 1542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:53539
I20260924 13:44:51.324873 1732 heartbeater.cc:503] Master 127.19.53.254:33747 was elected leader, sending a full tablet report...
Sep 24 13:44:51 dist-test-slave-tfm9 krb5kdc[955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257491, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for kudu/127.19.53.193@KRBTEST.COM
I20260924 13:44:51.333698 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.323643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.19.53.193:39533)
0924 13:44:51.324005 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:51.324042 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:51.324051 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:44:51.324082 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:51.324084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:51.324097 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:51.324154 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:51.324832 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.326290 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.327365 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.327831 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.327916 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.330337 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:51.330345 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:51.330346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:51.330353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:51.332481 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:51.332896 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:51.332898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:51.332898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:51.332957 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:51.333162 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:51.333163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:51.333164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:51.333297 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:51.333366 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:51.333550 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 13:44:51.333626 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":75,"threads_started":1}
I20260924 13:44:51.333717 1739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.323366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39533 (local address 127.0.0.1:39912)
0924 13:44:51.323797 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:51.323831 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:51.323892 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:51.324382 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:51.324386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:51.324687 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:51.324740 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.324742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.326368 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.326369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.327281 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.327284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.327931 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.327932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.328009 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.328547 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:51.328552 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:51.330229 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:51.332558 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:51.332564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:51.332567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:51.332811 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:51.333027 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:51.333028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:51.333028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:51.333094 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:51.333365 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:51.333371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:51.333422 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 13:44:51.333508 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 13:44:51.333616 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":94,"threads_started":1}
I20260924 13:44:51.334179 1665 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b9f47871f14e03b4c9e2df89fd2811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:44:51 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:39912
I20260924 13:44:51.334734 1741 tablet_replica.cc:330] stopping tablet replica
I20260924 13:44:51.334842 1741 raft_consensus.cc:2229] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:44:51.334900 1741 raft_consensus.cc:2258] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:44:51.335220 1741 ts_tablet_manager.cc:1905] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:44:51.337776 1741 ts_tablet_manager.cc:1918] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:44:51.337857 1741 log.cc:1196] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal/wals/e4b9f47871f14e03b4c9e2df89fd2811
I20260924 13:44:51.338102 1741 ts_tablet_manager.cc:1939] T e4b9f47871f14e03b4c9e2df89fd2811 P 1401dffdd88a4bc48db013e9bc33a15f: Deleting consensus metadata
I20260924 13:44:51.338636 1527 catalog_manager.cc:4841] TS 1401dffdd88a4bc48db013e9bc33a15f (127.19.53.193:39533): tablet e4b9f47871f14e03b4c9e2df89fd2811 (table test-table [id=a24f5c71874e4624a9e4053748d23ba3]) successfully deleted
W20260924 13:44:51.398849 1737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:51.399032 1737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:51.399055 1737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:51.400105 1737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:51.400136 1737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:51.400161 1737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:44:51.401539 1737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:43885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=40473
--tserver_master_addrs=127.19.53.254:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:51.401710 1737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:51.401880 1737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:51.406620 1746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:51.406623 1747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:51.406586 1749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:51.406802 1737 server_base.cc:1034] running on GCE node
I20260924 13:44:51.407040 1737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:51.407233 1737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:51.408372 1737 hybrid_clock.cc:648] HybridClock initialized: now 1790257491408352 us; error 40 us; skew 500 ppm
Sep 24 13:44:51 dist-test-slave-tfm9 krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257491, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:51.410668 1737 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:44:51.411036 1737 webserver.cc:460] Webserver started at http://127.19.53.194:40473/ using document root <none> and password file <none>
I20260924 13:44:51.411281 1737 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:51.411367 1737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:51.412665 1737 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:51.413229 1756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:51.413401 1737 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:44:51.413498 1737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "70ea85a903954bd8ace09deb103299cc"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "25d4b5230c20a68ce9e19c4edb2348b3"
server_key_iv: "514181abc6ed50bc0b65e5ecc4aacfc8"
server_key_version: "encryptionkey@0"
I20260924 13:44:51.413821 1737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:51.423166 1737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:51.425801 1737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:51.425949 1737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:51.426141 1737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:51.426390 1737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:44:51.426419 1737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:51.426440 1737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:44:51.426452 1737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:51.432602 1737 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:43885
I20260924 13:44:51.432643 1870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:43885 every 8 connection(s)
I20260924 13:44:51.432996 1737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:44:51.436792 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 1737
I20260924 13:44:51.437085 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:43169
--local_ip_for_outbound_sockets=127.19.53.195
--tserver_master_addrs=127.19.53.254:33747
--webserver_port=33393
--webserver_interface=127.19.53.195
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:44:51 dist-test-slave-tfm9 krb5kdc[955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257491, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:51.442869 1735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.433620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:46447 (local address 127.19.53.254:33747)
0924 13:44:51.433683 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:51.433685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:51.433883 (+ 198us) server_negotiation.cc:407] Connection header received
0924 13:44:51.434422 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:51.434423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:51.434435 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:51.434494 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:51.435389 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.435920 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.437013 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.437291 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.437396 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.439955 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:51.439956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:51.439957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:51.439963 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:51.441347 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:51.441851 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:51.441851 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:51.441852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:51.441898 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:51.442227 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:51.442228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:51.442228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:51.442342 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:51.442418 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:51.442678 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 13:44:51.442782 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 13:44:51.443163 1873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.433589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:46447)
0924 13:44:51.433738 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:51.433795 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:51.434336 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:51.434566 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:51.434569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:51.435087 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:51.435284 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.435288 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.436030 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.436032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.436913 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.436917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.437408 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.437410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.437548 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.438125 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:51.438133 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:51.439867 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:51.441430 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:51.441437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:51.441439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:51.441770 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:51.442021 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:51.442022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:51.442023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:51.442148 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:51.442409 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:51.442416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:51.442551 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 13:44:51.442659 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 13:44:51.442809 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260924 13:44:51.443696 1871 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33747
I20260924 13:44:51.443809 1871 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:51.444028 1871 heartbeater.cc:511] Master 127.19.53.254:33747 requested a full tablet report, sending...
I20260924 13:44:51.444392 1542 ts_manager.cc:194] Registered new tserver with Master: 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885)
I20260924 13:44:51.444774 1542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:46447
W20260924 13:44:51.518141 1874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:51.518357 1874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:51.518392 1874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:51.519531 1874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:51.519600 1874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:51.519641 1874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:44:51.521066 1874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:43169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=33393
--tserver_master_addrs=127.19.53.254:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:51.521330 1874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:51.521550 1874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:51.526510 1879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:51.526470 1880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:51.526476 1882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:51.526796 1874 server_base.cc:1034] running on GCE node
I20260924 13:44:51.526998 1874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:51.527240 1874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:51.528406 1874 hybrid_clock.cc:648] HybridClock initialized: now 1790257491528390 us; error 27 us; skew 500 ppm
Sep 24 13:44:51 dist-test-slave-tfm9 krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257491, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:51.531055 1874 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:44:51.531476 1874 webserver.cc:460] Webserver started at http://127.19.53.195:33393/ using document root <none> and password file <none>
I20260924 13:44:51.531699 1874 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:51.531786 1874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:51.533157 1874 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:51.533751 1889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:51.533989 1874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:51.534086 1874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "d80aae5f591e4114b26c16912088d070"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "832900b590bf9d5c82798e09ccade414"
server_key_iv: "4e450991ba43341707fd6629cb695c52"
server_key_version: "encryptionkey@0"
I20260924 13:44:51.534436 1874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:51.545219 1874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:51.547174 1874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:51.547334 1874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:51.547569 1874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:51.547883 1874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:44:51.547936 1874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:51.547971 1874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:44:51.547997 1874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:51.553570 1874 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:43169
I20260924 13:44:51.553637 2003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:43169 every 8 connection(s)
I20260924 13:44:51.554029 1874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:44:51 dist-test-slave-tfm9 krb5kdc[955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257491, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:44:51.562707 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 1874
I20260924 13:44:51.564220 1735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.554732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:48781 (local address 127.19.53.254:33747)
0924 13:44:51.554827 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:51.554830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:51.555008 (+ 178us) server_negotiation.cc:407] Connection header received
0924 13:44:51.555571 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:51.555572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:51.555585 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:51.555660 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:51.556629 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.557124 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.558490 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.558847 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.558927 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.561007 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:51.561009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:51.561009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:51.561015 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:51.562434 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:51.562867 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:51.562869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:51.562869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:51.562971 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:51.563307 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:51.563308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:51.563308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:51.563487 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:51.563548 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:51.563993 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 13:44:51.564068 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 13:44:51.564337 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.554716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:48781)
0924 13:44:51.554881 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:51.554926 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:51.555443 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:51.555732 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:51.555736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:51.556298 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:51.556507 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.556513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.557236 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.557237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.558409 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.558413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.558948 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.558948 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.559046 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.559561 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:51.559575 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:51.560923 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:51.562512 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:51.562518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:51.562520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:51.562762 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:51.563050 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:51.563051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:51.563051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:51.563226 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:51.563557 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:51.563564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:51.563773 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 13:44:51.563912 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 13:44:51.564080 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260924 13:44:51.564805 2004 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33747
I20260924 13:44:51.564913 2004 heartbeater.cc:464] Registering TS with master...
I20260924 13:44:51.565097 2004 heartbeater.cc:511] Master 127.19.53.254:33747 requested a full tablet report, sending...
I20260924 13:44:51.565516 1542 ts_manager.cc:194] Registered new tserver with Master: d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169)
I20260924 13:44:51.565959 1542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:48781
I20260924 13:44:51.569703 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.563171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.19.53.254:33747)
0924 13:44:51.563271 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:51.563273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:51.563367 (+ 94us) server_negotiation.cc:407] Connection header received
0924 13:44:51.563396 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:51.563396 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:51.563409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:51.563476 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:51.564206 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.564950 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.565849 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.566161 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.566227 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.567408 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:51.567409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:51.567410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:51.567416 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:51.568599 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:51.568945 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:51.568946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:51.568947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:51.569001 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:51.569209 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:51.569210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:51.569210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:51.569320 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:51.569499 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:51.569534 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 13:44:51.569605 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":46,"threads_started":1}
I20260924 13:44:51.570427 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:44:51.572759 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.570770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.19.53.254:33747)
0924 13:44:51.570803 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:51.570805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:51.570876 (+ 71us) server_negotiation.cc:407] Connection header received
0924 13:44:51.570931 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:51.570932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:51.570958 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:51.571016 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:44:51.571084 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.571527 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.572179 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.572368 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.572416 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.572529 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:44:51.572611 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:44:51.572664 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:51.572696 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 13:44:51.572712 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:44:51.573031 1542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:44:51.574800 1542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:44:51.576937 1665 tablet_service.cc:1467] Processing CreateTablet for tablet a358d4c8262847aa8d58cc7b3f65c598 (DEFAULT_TABLE table=test-table [id=158dbd6ba9854b1f836c7d48678dc527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:44:51.577118 1665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a358d4c8262847aa8d58cc7b3f65c598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:51.578740 1623 tablet_bootstrap.cc:492] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f: Bootstrap starting.
I20260924 13:44:51.579082 1623 tablet_bootstrap.cc:654] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:51.579669 1623 tablet_bootstrap.cc:492] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f: No bootstrap required, opened a new log
I20260924 13:44:51.579716 1623 ts_tablet_manager.cc:1397] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 13:44:51.580066 1623 raft_consensus.cc:348] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:51.580132 1623 raft_consensus.cc:374] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:51.580163 1623 raft_consensus.cc:729] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1401dffdd88a4bc48db013e9bc33a15f, State: Initialized, Role: FOLLOWER
I20260924 13:44:51.580222 1623 consensus_queue.cc:260] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:51.580345 1623 ts_tablet_manager.cc:1428] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:51.581192 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.576813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:43885 (local address 127.0.0.1:36790)
0924 13:44:51.577012 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:51.577023 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:51.577070 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:51.577432 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:51.577433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:51.577435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:51.577468 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.577469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.578722 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.578723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.579801 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.579805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.581016 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.581017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.581063 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.581076 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 13:44:51.581118 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 13:44:51.581147 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":95,"threads_started":1}
I20260924 13:44:51.581295 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.577055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36790 (local address 127.19.53.194:43885)
0924 13:44:51.577267 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:51.577271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:51.577282 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:44:51.577321 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:51.577323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:51.577339 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:51.577421 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:51.577517 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.578614 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.579891 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.580868 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.580926 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.580990 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:51.581153 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 13:44:51.581199 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
I20260924 13:44:51.581853 1804 tablet_service.cc:1467] Processing CreateTablet for tablet a358d4c8262847aa8d58cc7b3f65c598 (DEFAULT_TABLE table=test-table [id=158dbd6ba9854b1f836c7d48678dc527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:44:51.582227 1804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a358d4c8262847aa8d58cc7b3f65c598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:51.582255 1739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.576680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:43169 (local address 127.0.0.1:48234)
0924 13:44:51.576730 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:51.576742 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:51.576805 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:51.579092 (+ 2287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:51.579092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:51.579094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:51.579144 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.579145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.580328 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.580329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.581238 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.581241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.582011 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.582012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.582076 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.582080 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:44:51.582129 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 13:44:51.582177 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 13:44:51.582266 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.578761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.19.53.195:43169)
0924 13:44:51.578923 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:51.578928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:51.578938 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:44:51.578973 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:51.578975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:51.578990 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:51.579207 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:51.579264 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.580190 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.581333 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.581869 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.581990 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.582031 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:51.582164 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 13:44:51.582205 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260924 13:44:51.585024 2014 tablet_bootstrap.cc:492] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc: Bootstrap starting.
I20260924 13:44:51.585709 2014 tablet_bootstrap.cc:654] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:51.585630 1938 tablet_service.cc:1467] Processing CreateTablet for tablet a358d4c8262847aa8d58cc7b3f65c598 (DEFAULT_TABLE table=test-table [id=158dbd6ba9854b1f836c7d48678dc527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:44:51.586004 1938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a358d4c8262847aa8d58cc7b3f65c598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:44:51.586344 2014 log.cc:824] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:51.587083 2014 tablet_bootstrap.cc:492] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc: No bootstrap required, opened a new log
I20260924 13:44:51.587157 2014 ts_tablet_manager.cc:1397] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:44:51.589166 2014 raft_consensus.cc:348] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:51.589324 2014 raft_consensus.cc:374] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:51.589360 2014 raft_consensus.cc:729] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ea85a903954bd8ace09deb103299cc, State: Initialized, Role: FOLLOWER
I20260924 13:44:51.589468 2014 consensus_queue.cc:260] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:51.589512 2016 tablet_bootstrap.cc:492] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070: Bootstrap starting.
I20260924 13:44:51.589761 2014 ts_tablet_manager.cc:1428] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 13:44:51.589771 1871 heartbeater.cc:503] Master 127.19.53.254:33747 was elected leader, sending a full tablet report...
I20260924 13:44:51.590219 2016 tablet_bootstrap.cc:654] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070: Neither blocks nor log segments found. Creating new log.
I20260924 13:44:51.590827 2016 log.cc:824] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:51.591434 2016 tablet_bootstrap.cc:492] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070: No bootstrap required, opened a new log
I20260924 13:44:51.591481 2016 ts_tablet_manager.cc:1397] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:44:51.592646 2016 raft_consensus.cc:348] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:51.592767 2016 raft_consensus.cc:374] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:44:51.592787 2016 raft_consensus.cc:729] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80aae5f591e4114b26c16912088d070, State: Initialized, Role: FOLLOWER
I20260924 13:44:51.592885 2016 consensus_queue.cc:260] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:51.593091 2016 ts_tablet_manager.cc:1428] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:44:51.593137 2004 heartbeater.cc:503] Master 127.19.53.254:33747 was elected leader, sending a full tablet report...
W20260924 13:44:51.683557 1872 tablet.cc:2369] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:44:51.804557 2005 tablet.cc:2369] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:44:51.806771 1733 tablet.cc:2369] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:44:51.953689 2017 raft_consensus.cc:482] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:44:51.953824 2017 raft_consensus.cc:504] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:51.954298 2017 leader_election.cc:288] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169), 1401dffdd88a4bc48db013e9bc33a15f (127.19.53.193:39533)
I20260924 13:44:51.957319 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.954484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:43169 (local address 127.19.53.194:58481)
0924 13:44:51.954659 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:51.954672 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:51.954742 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:51.954917 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:51.954918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:51.954920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:51.955015 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.955017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.955686 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.955688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.956491 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.956493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.957053 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.957054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.957124 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.957128 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:44:51.957179 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 13:44:51.957222 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":9,"thread_start_us":64,"threads_started":1}
I20260924 13:44:51.957319 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.954593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:58481 (local address 127.19.53.195:43169)
0924 13:44:51.954639 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:51.954641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:51.954776 (+ 135us) server_negotiation.cc:407] Connection header received
0924 13:44:51.954841 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:51.954842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:51.954851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:51.954894 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:51.955092 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.955607 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.956587 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.956973 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.957029 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.957035 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:51.957204 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 13:44:51.957248 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:44:51.957398 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.954552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:41203 (local address 127.19.53.193:39533)
0924 13:44:51.954708 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:51.954710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:51.954719 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:44:51.954804 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:51.954805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:51.954832 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:51.954929 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:44:51.955121 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.955631 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.956615 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.957027 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.957104 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.957185 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:51.957274 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 13:44:51.957318 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":46,"threads_started":1}
I20260924 13:44:51.957460 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:51.954480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39533 (local address 127.19.53.194:41203)
0924 13:44:51.954633 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:51.954644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:51.954700 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:51.954947 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:51.954948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:51.954950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:44:51.955031 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.955032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.955740 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.955741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.956505 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:51.956507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:51.957120 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:51.957121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:51.957200 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:51.957204 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:44:51.957267 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 13:44:51.957304 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260924 13:44:51.957760 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a358d4c8262847aa8d58cc7b3f65c598" candidate_uuid: "70ea85a903954bd8ace09deb103299cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1401dffdd88a4bc48db013e9bc33a15f" is_pre_election: true
I20260924 13:44:51.957958 1685 raft_consensus.cc:2454] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70ea85a903954bd8ace09deb103299cc in term 0.
I20260924 13:44:51.957930 1958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a358d4c8262847aa8d58cc7b3f65c598" candidate_uuid: "70ea85a903954bd8ace09deb103299cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d80aae5f591e4114b26c16912088d070" is_pre_election: true
I20260924 13:44:51.958065 1958 raft_consensus.cc:2454] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70ea85a903954bd8ace09deb103299cc in term 0.
I20260924 13:44:51.958196 1758 leader_election.cc:302] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1401dffdd88a4bc48db013e9bc33a15f, 70ea85a903954bd8ace09deb103299cc; no voters:
I20260924 13:44:51.958379 2017 raft_consensus.cc:2781] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:44:51.958425 2017 raft_consensus.cc:482] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:44:51.958441 2017 raft_consensus.cc:3037] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:51.959101 2017 raft_consensus.cc:504] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:51.959236 2017 leader_election.cc:288] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [CANDIDATE]: Term 1 election: Requested vote from peers d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169), 1401dffdd88a4bc48db013e9bc33a15f (127.19.53.193:39533)
I20260924 13:44:51.959379 1958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a358d4c8262847aa8d58cc7b3f65c598" candidate_uuid: "70ea85a903954bd8ace09deb103299cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d80aae5f591e4114b26c16912088d070"
I20260924 13:44:51.959395 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a358d4c8262847aa8d58cc7b3f65c598" candidate_uuid: "70ea85a903954bd8ace09deb103299cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1401dffdd88a4bc48db013e9bc33a15f"
I20260924 13:44:51.959460 1685 raft_consensus.cc:3037] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:51.959460 1958 raft_consensus.cc:3037] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:44:51.960276 1685 raft_consensus.cc:2454] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70ea85a903954bd8ace09deb103299cc in term 1.
I20260924 13:44:51.960279 1958 raft_consensus.cc:2454] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70ea85a903954bd8ace09deb103299cc in term 1.
I20260924 13:44:51.960451 1760 leader_election.cc:302] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ea85a903954bd8ace09deb103299cc, d80aae5f591e4114b26c16912088d070; no voters:
I20260924 13:44:51.960561 2017 raft_consensus.cc:2781] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:44:51.960695 2017 raft_consensus.cc:686] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [term 1 LEADER]: Becoming Leader. State: Replica: 70ea85a903954bd8ace09deb103299cc, State: Running, Role: LEADER
I20260924 13:44:51.960799 2017 consensus_queue.cc:237] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:51.961711 1541 catalog_manager.cc:5495] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 70ea85a903954bd8ace09deb103299cc (127.19.53.194). New cstate: current_term: 1 leader_uuid: "70ea85a903954bd8ace09deb103299cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } health_report { overall_health: UNKNOWN } } }
I20260924 13:44:52.021329 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:52.018889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.19.53.194:43885)
0924 13:44:52.018956 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:52.018958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:52.018968 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:44:52.019083 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:52.019085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:52.019103 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:52.019192 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:44:52.019320 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:52.019823 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:52.020540 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:52.020736 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:52.020809 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:52.020983 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:44:52.021090 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:44:52.021186 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:52.021210 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 13:44:52.021239 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:44:52.023252 1958 raft_consensus.cc:1261] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070 [term 1 FOLLOWER]: Refusing update from remote peer 70ea85a903954bd8ace09deb103299cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:44:52.023286 1685 raft_consensus.cc:1261] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Refusing update from remote peer 70ea85a903954bd8ace09deb103299cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:44:52.023625 2017 consensus_queue.cc:1035] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:44:52.023788 2023 consensus_queue.cc:1035] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:44:52.025769 2025 mvcc.cc:205] Tried to move back new op lower bound from 7332894687324311552 to 7332894687071997952. Current Snapshot: MvccSnapshot[applied={T|T < 7332894687324311552}]
I20260924 13:44:52.025810 2026 mvcc.cc:205] Tried to move back new op lower bound from 7332894687324311552 to 7332894687071997952. Current Snapshot: MvccSnapshot[applied={T|T < 7332894687324311552}]
I20260924 13:44:52.028577 1541 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:44:52.028687 1541 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:44:52.029439 1541 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Sending DeleteTablet for 3 replicas of tablet a358d4c8262847aa8d58cc7b3f65c598
I20260924 13:44:52.029773 1665 tablet_service.cc:1514] Processing DeleteTablet for tablet a358d4c8262847aa8d58cc7b3f65c598 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:44:52 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:39912
I20260924 13:44:52.029780 1804 tablet_service.cc:1514] Processing DeleteTablet for tablet a358d4c8262847aa8d58cc7b3f65c598 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:44:52 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:36790
I20260924 13:44:52.029807 1938 tablet_service.cc:1514] Processing DeleteTablet for tablet a358d4c8262847aa8d58cc7b3f65c598 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:44:52 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:48234
I20260924 13:44:52.030088 2037 tablet_replica.cc:330] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070: stopping tablet replica
I20260924 13:44:52.030115 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 1580
I20260924 13:44:52.030195 2037 raft_consensus.cc:2229] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:44:52.030305 2037 raft_consensus.cc:2258] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:44:52.030813 2037 ts_tablet_manager.cc:1905] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:44:52.032836 2036 tablet_replica.cc:330] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc: stopping tablet replica
I20260924 13:44:52.032936 2036 raft_consensus.cc:2229] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:44:52.033039 2037 ts_tablet_manager.cc:1918] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:44:52.033056 2036 raft_consensus.cc:2258] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:44:52.033157 2037 log.cc:1196] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/a358d4c8262847aa8d58cc7b3f65c598
I20260924 13:44:52.033460 2036 ts_tablet_manager.cc:1905] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:44:52.033731 2037 ts_tablet_manager.cc:1939] T a358d4c8262847aa8d58cc7b3f65c598 P d80aae5f591e4114b26c16912088d070: Deleting consensus metadata
I20260924 13:44:52.034260 1529 catalog_manager.cc:4841] TS d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169): tablet a358d4c8262847aa8d58cc7b3f65c598 (table test-table [id=158dbd6ba9854b1f836c7d48678dc527]) successfully deleted
I20260924 13:44:52.035523 2036 ts_tablet_manager.cc:1918] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:44:52.035597 2036 log.cc:1196] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/a358d4c8262847aa8d58cc7b3f65c598
I20260924 13:44:52.035858 2036 ts_tablet_manager.cc:1939] T a358d4c8262847aa8d58cc7b3f65c598 P 70ea85a903954bd8ace09deb103299cc: Deleting consensus metadata
I20260924 13:44:52.036356 1529 catalog_manager.cc:4841] TS 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885): tablet a358d4c8262847aa8d58cc7b3f65c598 (table test-table [id=158dbd6ba9854b1f836c7d48678dc527]) successfully deleted
W20260924 13:44:52.037607 1527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:39533 (error 108)
I20260924 13:44:52.037859 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 1737
I20260924 13:44:52.038468 1739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:52.038153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39533 (local address 127.0.0.1:39918)
0924 13:44:52.038386 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:52.038430 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:39533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
W20260924 13:44:52.038612 1527 catalog_manager.cc:4584] TS 1401dffdd88a4bc48db013e9bc33a15f (127.19.53.193:39533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a358d4c8262847aa8d58cc7b3f65c598: Network error: Client connection negotiation failed: client connection to 127.19.53.193:39533: connect: Connection refused (error 111)
I20260924 13:44:52.043788 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 1874
I20260924 13:44:52.047930 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 1509
I20260924 13:44:52.053043 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33747
--webserver_interface=127.19.53.254
--webserver_port=45217
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:33747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:44:52.132916 2039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:52.133116 2039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:52.133138 2039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:52.134230 2039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:44:52.134263 2039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:52.134279 2039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:52.134290 2039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:44:52.134303 2039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:44:52.135668 2039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:33747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=45217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:52.135852 2039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:52.136014 2039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:52.140743 2047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:52.140754 2045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:52.140758 2044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:52.141054 2039 server_base.cc:1034] running on GCE node
I20260924 13:44:52.141268 2039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:52.141517 2039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:52.142657 2039 hybrid_clock.cc:648] HybridClock initialized: now 1790257492142645 us; error 37 us; skew 500 ppm
Sep 24 13:44:52 dist-test-slave-tfm9 krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257492, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:52.144968 2039 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:44:52.145334 2039 webserver.cc:460] Webserver started at http://127.19.53.254:45217/ using document root <none> and password file <none>
I20260924 13:44:52.145521 2039 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:52.145562 2039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:52.146872 2039 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:52.147531 2055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:52.147684 2039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:52.147760 2039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "b14fe8c6eedf22e037258a9644f91dfd"
server_key_iv: "e299cce4fb5251a781889789bd34d0a7"
server_key_version: "encryptionkey@0"
I20260924 13:44:52.148067 2039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:52.159929 2039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:52.161640 2039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:52.161754 2039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:52.165421 2039 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:33747
I20260924 13:44:52.165458 2108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:33747 every 8 connection(s)
I20260924 13:44:52.165836 2039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:44:52.167412 2109 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:44:52.167896 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 2039
I20260924 13:44:52.168165 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:39533
--local_ip_for_outbound_sockets=127.19.53.193
--tserver_master_addrs=127.19.53.254:33747
--webserver_port=41545
--webserver_interface=127.19.53.193
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:44:52.168287 2109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Bootstrap starting.
I20260924 13:44:52.176961 2109 log.cc:824] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:52.179286 2109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:44:52.179580 2109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f: Bootstrap complete.
I20260924 13:44:52.181824 2109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } }
I20260924 13:44:52.182147 2109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e420cb8232a4df7b40f7ff2cfe2b51f, State: Initialized, Role: FOLLOWER
I20260924 13:44:52.182353 2109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } }
I20260924 13:44:52.182466 2109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:44:52.182518 2109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:44:52.182564 2109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 2 FOLLOWER]: Advancing to term 3
I20260924 13:44:52.183568 2109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } }
I20260924 13:44:52.183681 2109 leader_election.cc:302] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e420cb8232a4df7b40f7ff2cfe2b51f; no voters:
I20260924 13:44:52.183885 2109 leader_election.cc:288] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 13:44:52.183907 2113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 3 FOLLOWER]: Leader election won for term 3
I20260924 13:44:52.184017 2113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [term 3 LEADER]: Becoming Leader. State: Replica: 8e420cb8232a4df7b40f7ff2cfe2b51f, State: Running, Role: LEADER
I20260924 13:44:52.184118 2113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } }
I20260924 13:44:52.184156 2109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:44:52.184384 2115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } } }
I20260924 13:44:52.184474 2115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:52.184624 2114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e420cb8232a4df7b40f7ff2cfe2b51f. Latest consensus state: current_term: 3 leader_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e420cb8232a4df7b40f7ff2cfe2b51f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33747 } } }
I20260924 13:44:52.184676 2114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e420cb8232a4df7b40f7ff2cfe2b51f [sys.catalog]: This master's current role is: LEADER
I20260924 13:44:52.184777 2117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:44:52.185719 2117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:44:52.185896 2117 catalog_manager.cc:1259] Loaded cluster ID: 75729e012f9b4279b248f14258d7cf02
I20260924 13:44:52.186149 2117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 13:44:52.186367 2117 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2baa6f99ad google::LogMessage::Fail() at ??:0
@ 0x7f2baa6fc88a google::LogMessage::SendToLog() at ??:0
@ 0x7f2baa6f94de google::LogMessage::Flush() at ??:0
@ 0x7f2baa6faa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2bb261a0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f2bb26238b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2bac5c70d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2bac5bc954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2bad4206db start_thread at ??:0
@ 0x7f2ba91f761f clone at ??:0
W20260924 13:44:52.264889 2111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:52.265079 2111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:52.265101 2111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:52.266310 2111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:52.266351 2111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:52.266377 2111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:44:52.267798 2111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:39533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=41545
--tserver_master_addrs=127.19.53.254:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:52.268033 2111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:52.268275 2111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:52.274116 2145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:52.274123 2149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:52.274123 2147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:52.274456 2111 server_base.cc:1034] running on GCE node
I20260924 13:44:52.274657 2111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:52.274904 2111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:52.276063 2111 hybrid_clock.cc:648] HybridClock initialized: now 1790257492276051 us; error 34 us; skew 500 ppm
Sep 24 13:44:52 dist-test-slave-tfm9 krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257492, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:52.278579 2111 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:44:52.279071 2111 webserver.cc:460] Webserver started at http://127.19.53.193:41545/ using document root <none> and password file <none>
I20260924 13:44:52.279336 2111 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:52.279382 2111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:52.280736 2111 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:52.281411 2156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:52.281620 2111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:52.281729 2111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "1401dffdd88a4bc48db013e9bc33a15f"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "03687ec81db6c5743550731ace883b03"
server_key_iv: "25778ed2ec4a1d25e7ed709998600537"
server_key_version: "encryptionkey@0"
I20260924 13:44:52.282191 2111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:52.315850 2111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:52.318476 2111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:52.318638 2111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:52.318936 2111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:52.319473 2163 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 13:44:52.320993 2111 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 13:44:52.321050 2111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 13:44:52.321084 2111 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 13:44:52.321945 2111 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 13:44:52.321987 2111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260924 13:44:52.322111 2163 tablet_bootstrap.cc:492] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f: Bootstrap starting.
I20260924 13:44:52.329036 2111 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:39533
I20260924 13:44:52.329278 2271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:39533 every 8 connection(s)
I20260924 13:44:52.329504 2111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:44:52.332922 2163 log.cc:824] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:44:52.334100 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 2111
I20260924 13:44:52.334410 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:43885
--local_ip_for_outbound_sockets=127.19.53.194
--tserver_master_addrs=127.19.53.254:33747
--webserver_port=40473
--webserver_interface=127.19.53.194
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:44:52.335681 2163 tablet_bootstrap.cc:492] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:44:52.335978 2163 tablet_bootstrap.cc:492] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f: Bootstrap complete.
I20260924 13:44:52.336912 2163 ts_tablet_manager.cc:1397] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260924 13:44:52.338027 2163 raft_consensus.cc:348] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:52.338420 2163 raft_consensus.cc:729] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1401dffdd88a4bc48db013e9bc33a15f, State: Initialized, Role: FOLLOWER
I20260924 13:44:52.338590 2163 consensus_queue.cc:260] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:52.338874 2163 ts_tablet_manager.cc:1428] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260924 13:44:52.374581 2274 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:44:52.330219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:57241)
0924 13:44:52.330437 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:52.330486 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:52.331102 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:52.374381 (+ 43279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":108,"threads_started":1}
W20260924 13:44:52.374805 2272 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:33747 (0 consecutive failures): Network error: Failed to ping master at 127.19.53.254:33747: Client connection negotiation failed: client connection to 127.19.53.254:33747: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 13:44:52.375152 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:52.375026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:52199)
0924 13:44:52.375082 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:52.375121 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 13:44:52.375552 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:52.375461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:55791)
0924 13:44:52.375516 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:52.375537 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 13:44:52.375666 2272 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:44:52.419361 2276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:52.419543 2276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:52.419566 2276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:52.420624 2276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:52.420656 2276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:52.420691 2276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:44:52.422077 2276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:43885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=40473
--tserver_master_addrs=127.19.53.254:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:52.422238 2276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:52.422399 2276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:52.427282 2282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:52.427358 2283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:52.427328 2285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:52.427685 2276 server_base.cc:1034] running on GCE node
I20260924 13:44:52.427915 2276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:52.428155 2276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:52.429318 2276 hybrid_clock.cc:648] HybridClock initialized: now 1790257492429295 us; error 43 us; skew 500 ppm
Sep 24 13:44:52 dist-test-slave-tfm9 krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257492, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:52.431923 2276 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:44:52.432334 2276 webserver.cc:460] Webserver started at http://127.19.53.194:40473/ using document root <none> and password file <none>
I20260924 13:44:52.432572 2276 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:52.432639 2276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:52.433899 2276 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:52.434527 2292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:52.434686 2276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:52.434780 2276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "70ea85a903954bd8ace09deb103299cc"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "25d4b5230c20a68ce9e19c4edb2348b3"
server_key_iv: "514181abc6ed50bc0b65e5ecc4aacfc8"
server_key_version: "encryptionkey@0"
I20260924 13:44:52.435124 2276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:52.452435 2276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:52.454249 2276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:52.454398 2276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:52.454666 2276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:52.454977 2276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:44:52.455032 2276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:52.455077 2276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:44:52.455101 2276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:52.460889 2276 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:43885
I20260924 13:44:52.460934 2406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:43885 every 8 connection(s)
I20260924 13:44:52.461270 2276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:44:52.461530 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 2276
I20260924 13:44:52.461844 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:43169
--local_ip_for_outbound_sockets=127.19.53.195
--tserver_master_addrs=127.19.53.254:33747
--webserver_port=33393
--webserver_interface=127.19.53.195
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:44:52.462648 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:52.462265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:42307)
0924 13:44:52.462433 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:52.462511 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
W20260924 13:44:52.463155 2407 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:33747 (0 consecutive failures): Network error: Failed to ping master at 127.19.53.254:33747: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
I20260924 13:44:52.463514 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:52.463391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:51517)
0924 13:44:52.463447 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:52.463485 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 13:44:52.463891 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:52.463806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:54239)
0924 13:44:52.463855 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:52.463877 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 13:44:52.464035 2407 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:44:52.541347 2410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:44:52.541524 2410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:44:52.541545 2410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:44:52.542631 2410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:44:52.542670 2410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:44:52.542716 2410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:44:52.544060 2410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:43169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=33393
--tserver_master_addrs=127.19.53.254:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:44:52.544292 2410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:44:52.544504 2410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:44:52.549517 2418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:44:52.549546 2416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:52.549583 2410 server_base.cc:1034] running on GCE node
W20260924 13:44:52.549561 2415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:44:52.549968 2410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:44:52.550202 2410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:44:52.551349 2410 hybrid_clock.cc:648] HybridClock initialized: now 1790257492551338 us; error 28 us; skew 500 ppm
Sep 24 13:44:52 dist-test-slave-tfm9 krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257492, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:44:52.553617 2410 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:44:52.554129 2410 webserver.cc:460] Webserver started at http://127.19.53.195:33393/ using document root <none> and password file <none>
I20260924 13:44:52.554361 2410 fs_manager.cc:359] Metadata directory not provided
I20260924 13:44:52.554406 2410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:44:52.555672 2410 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:44:52.556213 2425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:52.556391 2410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:44:52.556470 2410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "d80aae5f591e4114b26c16912088d070"
format_stamp: "Formatted at 2026-09-24 13:44:50 on dist-test-slave-tfm9"
server_key: "832900b590bf9d5c82798e09ccade414"
server_key_iv: "4e450991ba43341707fd6629cb695c52"
server_key_version: "encryptionkey@0"
I20260924 13:44:52.556783 2410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:44:52.582908 2410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:44:52.585216 2410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:44:52.585331 2410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:44:52.585562 2410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:44:52.585857 2410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:44:52.585888 2410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:52.585930 2410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:44:52.585944 2410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:44:52.591557 2410 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:43169
I20260924 13:44:52.591607 2539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:43169 every 8 connection(s)
I20260924 13:44:52.591928 2410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:44:52.592917 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:52.592549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:47495)
0924 13:44:52.592705 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:52.592783 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
W20260924 13:44:52.593300 2540 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:33747 (0 consecutive failures): Network error: Failed to ping master at 127.19.53.254:33747: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
I20260924 13:44:52.593549 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:52.593460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:41221)
0924 13:44:52.593503 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:52.593525 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 13:44:52.593792 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:52.593717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:34227)
0924 13:44:52.593754 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:52.593774 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 13:44:52.593890 2540 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 13:44:52.597285 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 2410
I20260924 13:44:53.376173 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:53.375962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:40075)
0924 13:44:53.376078 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:53.376111 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260924 13:44:53.464566 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:53.464369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:34839)
0924 13:44:53.464482 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:53.464514 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260924 13:44:53.593137 2546 raft_consensus.cc:482] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:44:53.593298 2546 raft_consensus.cc:504] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:53.593662 2546 leader_election.cc:288] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169), 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885)
I20260924 13:44:53.594353 2550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:53.594181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:33555)
0924 13:44:53.594282 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:53.594315 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
Sep 24 13:44:53 dist-test-slave-tfm9 krb5kdc[955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257492, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.194@KRBTEST.COM
Sep 24 13:44:53 dist-test-slave-tfm9 krb5kdc[955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257492, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.195@KRBTEST.COM
I20260924 13:44:53.603940 2549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:53.594089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:43891 (local address 127.19.53.194:43885)
0924 13:44:53.594224 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:53.594230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:53.594254 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:44:53.594753 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:53.594756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:53.594793 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:53.594945 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:53.595597 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:53.596467 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:53.597458 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:53.597859 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:53.598010 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:53.600206 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:53.600210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:53.600211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:53.600222 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:53.602221 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:53.602740 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:53.602742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:53.602743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:53.602833 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:53.603111 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:53.603112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:53.603112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:53.603341 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:53.603428 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:53.603700 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 13:44:53.603833 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":50,"threads_started":1}
I20260924 13:44:53.603927 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:53.593996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:43885 (local address 127.19.53.193:43891)
0924 13:44:53.594106 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:53.594117 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:53.594173 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:53.595073 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:53.595074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:53.595285 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:53.595477 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:53.595482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:53.596576 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:53.596577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:53.597366 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:53.597368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:53.597962 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:53.597963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:53.598094 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:53.598665 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:53.598672 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:53.600102 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:53.602315 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:53.602339 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:53.602341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:53.602623 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:53.602915 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:53.602916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:53.602916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:53.603042 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:53.603424 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:53.603430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:53.603547 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 13:44:53.603650 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 13:44:53.603830 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":79,"thread_start_us":59,"threads_started":1}
I20260924 13:44:53.603955 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:53.593950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:52583 (local address 127.19.53.195:43169)
0924 13:44:53.594139 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 13:44:53.594151 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:44:53.594175 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:44:53.594715 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:44:53.594718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:44:53.594782 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:44:53.594945 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:44:53.595594 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:53.596455 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:53.597457 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:44:53.597958 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:44:53.598060 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:53.600656 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:44:53.600663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:44:53.600664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:44:53.600672 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:44:53.602314 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:53.602749 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:53.602752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:53.602752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:53.602840 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:44:53.603117 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:44:53.603118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:44:53.603118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:44:53.603345 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:44:53.603438 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 13:44:53.603704 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 13:44:53.603826 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260924 13:44:53.604122 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:53.593830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:43169 (local address 127.19.53.193:52583)
0924 13:44:53.593895 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:53.593906 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:44:53.593980 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:44:53.594990 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:44:53.594993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:44:53.595292 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:44:53.595477 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:53.595482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:53.596553 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:53.596554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:53.597367 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:44:53.597370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:44:53.598036 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:44:53.598038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:44:53.598111 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:44:53.598733 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:44:53.598738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:44:53.600546 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:44:53.602417 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:53.602418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:53.602419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:53.602638 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:53.602924 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:44:53.602924 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:44:53.602925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:44:53.603048 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:44:53.603420 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:44:53.603427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:44:53.603547 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 13:44:53.603651 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 13:44:53.603911 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":155}
W20260924 13:44:53.604825 2160 leader_election.cc:341] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169): Not found: Tablet not found: a358d4c8262847aa8d58cc7b3f65c598
W20260924 13:44:53.605345 2160 leader_election.cc:341] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885): Not found: Tablet not found: a358d4c8262847aa8d58cc7b3f65c598
I20260924 13:44:53.605414 2160 leader_election.cc:302] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1401dffdd88a4bc48db013e9bc33a15f; no voters: 70ea85a903954bd8ace09deb103299cc, d80aae5f591e4114b26c16912088d070
I20260924 13:44:53.605527 2546 raft_consensus.cc:2726] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:44:54.376848 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:54.376618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:33767)
0924 13:44:54.376768 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:54.376804 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260924 13:44:54.465220 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:54.465032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:41981)
0924 13:44:54.465150 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:54.465182 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260924 13:44:54.594902 2553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:54.594704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:45017)
0924 13:44:54.594829 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:54.594865 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260924 13:44:55.377599 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:55.377338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:54561)
0924 13:44:55.377493 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:55.377536 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":62,"threads_started":1}
I20260924 13:44:55.465977 2555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:55.465700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:33295)
0924 13:44:55.465831 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:55.465895 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260924 13:44:55.595556 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:55.595344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:59961)
0924 13:44:55.595486 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:55.595518 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260924 13:44:55.662647 2557 raft_consensus.cc:482] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:44:55.662745 2557 raft_consensus.cc:504] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:55.662914 2557 leader_election.cc:288] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169), 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885)
W20260924 13:44:55.663396 2160 leader_election.cc:341] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885): Not found: Tablet not found: a358d4c8262847aa8d58cc7b3f65c598
W20260924 13:44:55.663501 2160 leader_election.cc:341] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169): Not found: Tablet not found: a358d4c8262847aa8d58cc7b3f65c598
I20260924 13:44:55.663537 2160 leader_election.cc:302] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1401dffdd88a4bc48db013e9bc33a15f; no voters: 70ea85a903954bd8ace09deb103299cc, d80aae5f591e4114b26c16912088d070
I20260924 13:44:55.663651 2557 raft_consensus.cc:2726] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:44:56.378366 2558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:56.378118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:56653)
0924 13:44:56.378260 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:56.378296 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":56,"threads_started":1}
I20260924 13:44:56.466652 2559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:56.466413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:35973)
0924 13:44:56.466572 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:56.466611 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260924 13:44:56.596168 2560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:56.595995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:36427)
0924 13:44:56.596102 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:56.596131 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260924 13:44:57.379091 2561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:57.378865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:47117)
0924 13:44:57.379005 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:57.379042 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260924 13:44:57.467329 2562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:57.467103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:58075)
0924 13:44:57.467243 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:57.467275 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260924 13:44:57.596872 2563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:57.596631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:42793)
0924 13:44:57.596791 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:57.596830 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":60,"threads_started":1}
I20260924 13:44:58.192291 2564 raft_consensus.cc:482] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:44:58.192389 2564 raft_consensus.cc:504] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:44:58.192510 2564 leader_election.cc:288] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169), 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885)
W20260924 13:44:58.193005 2160 leader_election.cc:341] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169): Not found: Tablet not found: a358d4c8262847aa8d58cc7b3f65c598
W20260924 13:44:58.193099 2160 leader_election.cc:341] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885): Not found: Tablet not found: a358d4c8262847aa8d58cc7b3f65c598
I20260924 13:44:58.193148 2160 leader_election.cc:302] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1401dffdd88a4bc48db013e9bc33a15f; no voters: 70ea85a903954bd8ace09deb103299cc, d80aae5f591e4114b26c16912088d070
I20260924 13:44:58.193248 2564 raft_consensus.cc:2726] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:44:58.379745 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:58.379528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:51093)
0924 13:44:58.379661 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:58.379698 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260924 13:44:58.468627 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:58.468330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:42957)
0924 13:44:58.468526 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:58.468582 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":96,"threads_started":1}
I20260924 13:44:58.597523 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:58.597306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:44745)
0924 13:44:58.597446 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:58.597481 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260924 13:44:59.380443 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:59.380216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:58803)
0924 13:44:59.380344 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:59.380385 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260924 13:44:59.469341 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:59.469141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:54375)
0924 13:44:59.469268 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:59.469302 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260924 13:44:59.598271 2570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:44:59.598077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:37087)
0924 13:44:59.598191 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 13:44:59.598231 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260924 13:45:00.381201 2571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:00.380929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:52537)
0924 13:45:00.381107 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:00.381150 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":117,"threads_started":1}
I20260924 13:45:00.470166 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:00.469874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:57875)
0924 13:45:00.470070 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:00.470106 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":125,"threads_started":1}
I20260924 13:45:00.598850 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:00.598650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:45481)
0924 13:45:00.598768 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:00.598804 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260924 13:45:01.158326 2574 raft_consensus.cc:482] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:45:01.158439 2574 raft_consensus.cc:504] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:45:01.158600 2574 leader_election.cc:288] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169), 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885)
W20260924 13:45:01.158946 2160 leader_election.cc:341] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169): Not found: Tablet not found: a358d4c8262847aa8d58cc7b3f65c598
W20260924 13:45:01.159013 2160 leader_election.cc:341] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885): Not found: Tablet not found: a358d4c8262847aa8d58cc7b3f65c598
I20260924 13:45:01.159034 2160 leader_election.cc:302] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1401dffdd88a4bc48db013e9bc33a15f; no voters: 70ea85a903954bd8ace09deb103299cc, d80aae5f591e4114b26c16912088d070
I20260924 13:45:01.159099 2574 raft_consensus.cc:2726] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:45:01.382077 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:01.381819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:41533)
0924 13:45:01.381993 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:01.382030 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":60,"threads_started":1}
I20260924 13:45:01.470806 2576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:01.470561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:55383)
0924 13:45:01.470700 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:01.470743 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":57,"threads_started":1}
I20260924 13:45:01.599509 2577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:01.599274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:46271)
0924 13:45:01.599416 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:01.599453 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260924 13:45:02.382764 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:02.382565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:45837)
0924 13:45:02.382681 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:02.382717 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260924 13:45:02.471567 2579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:02.471339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:55985)
0924 13:45:02.471479 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:02.471520 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260924 13:45:02.600163 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:02.599927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:48743)
0924 13:45:02.600073 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:02.600117 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":59,"threads_started":1}
I20260924 13:45:02.996632 2581 raft_consensus.cc:482] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:45:02.996724 2581 raft_consensus.cc:504] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:45:02.996892 2581 leader_election.cc:288] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169), 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885)
W20260924 13:45:02.997380 2160 leader_election.cc:341] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169): Not found: Tablet not found: a358d4c8262847aa8d58cc7b3f65c598
W20260924 13:45:02.997478 2160 leader_election.cc:341] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885): Not found: Tablet not found: a358d4c8262847aa8d58cc7b3f65c598
I20260924 13:45:02.997517 2160 leader_election.cc:302] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1401dffdd88a4bc48db013e9bc33a15f; no voters: 70ea85a903954bd8ace09deb103299cc, d80aae5f591e4114b26c16912088d070
I20260924 13:45:02.997615 2581 raft_consensus.cc:2726] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:45:03.383481 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:03.383244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:52085)
0924 13:45:03.383398 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:03.383433 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260924 13:45:03.472256 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:03.472051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:46085)
0924 13:45:03.472164 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:03.472213 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260924 13:45:03.600991 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:03.600713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:33255)
0924 13:45:03.600882 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:03.600923 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":100,"threads_started":1}
I20260924 13:45:04.384299 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:04.384072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:55947)
0924 13:45:04.384215 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:04.384251 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260924 13:45:04.473086 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:04.472861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:43777)
0924 13:45:04.473005 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:04.473042 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260924 13:45:04.601681 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:04.601472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:53367)
0924 13:45:04.601600 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:04.601633 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260924 13:45:05.384987 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:05.384792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:51921)
0924 13:45:05.384908 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:05.384941 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260924 13:45:05.473739 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:05.473535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:48885)
0924 13:45:05.473664 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:05.473697 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260924 13:45:05.602466 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:05.602249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:43107)
0924 13:45:05.602384 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:05.602422 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260924 13:45:05.784569 2591 raft_consensus.cc:482] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:45:05.784662 2591 raft_consensus.cc:504] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80aae5f591e4114b26c16912088d070" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 43169 } } peers { permanent_uuid: "70ea85a903954bd8ace09deb103299cc" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43885 } } peers { permanent_uuid: "1401dffdd88a4bc48db013e9bc33a15f" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39533 } }
I20260924 13:45:05.784842 2591 leader_election.cc:288] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169), 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885)
W20260924 13:45:05.785291 2160 leader_election.cc:341] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d80aae5f591e4114b26c16912088d070 (127.19.53.195:43169): Not found: Tablet not found: a358d4c8262847aa8d58cc7b3f65c598
W20260924 13:45:05.785372 2160 leader_election.cc:341] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70ea85a903954bd8ace09deb103299cc (127.19.53.194:43885): Not found: Tablet not found: a358d4c8262847aa8d58cc7b3f65c598
I20260924 13:45:05.785388 2160 leader_election.cc:302] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1401dffdd88a4bc48db013e9bc33a15f; no voters: 70ea85a903954bd8ace09deb103299cc, d80aae5f591e4114b26c16912088d070
I20260924 13:45:05.785459 2591 raft_consensus.cc:2726] T a358d4c8262847aa8d58cc7b3f65c598 P 1401dffdd88a4bc48db013e9bc33a15f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:45:06.385785 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:06.385532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:36885)
0924 13:45:06.385692 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:06.385734 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":57,"threads_started":1}
I20260924 13:45:06.474328 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:06.474106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:57105)
0924 13:45:06.474245 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:06.474281 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260924 13:45:06.603147 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:06.602920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:44459)
0924 13:45:06.603056 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:06.603099 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260924 13:45:07.386632 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:07.386407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.193:53385)
0924 13:45:07.386542 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:07.386583 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260924 13:45:07.474982 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:07.474788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.194:51113)
0924 13:45:07.474907 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:07.474938 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260924 13:45:07.598832 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 2111
I20260924 13:45:07.603757 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 2276
I20260924 13:45:07.603823 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:07.603636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33747 (local address 127.19.53.195:35815)
0924 13:45:07.603755 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:07.603783 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260924 13:45:07.608052 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 2410
2026-09-24T13:45:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18481 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:45:07 dist-test-slave-tfm9 krb5kdc[2600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:45:07 dist-test-slave-tfm9 krb5kdc[2600](info): set up 2 sockets
Sep 24 13:45:07 dist-test-slave-tfm9 krb5kdc[2600](info): commencing operation
krb5kdc: starting...
W20260924 13:45:08.643558 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:45:08 dist-test-slave-tfm9 krb5kdc[2600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:45:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:45:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:45:08.760941 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:38623
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:45:08.841027 2613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:45:08.841219 2613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:45:08.841241 2613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:45:08.842375 2613 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 13:45:08.842411 2613 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 13:45:08.842424 2613 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 13:45:08.842470 2613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:45:08.842489 2613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:45:08.842512 2613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:45:08.842537 2613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:45:08.842561 2613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:45:08.843974 2613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:38623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:45:08.844199 2613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:45:08.844440 2613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:45:08.849677 2619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:08.849718 2613 server_base.cc:1034] running on GCE node
W20260924 13:45:08.849700 2621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:08.849696 2618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:08.850124 2613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:45:08.850410 2613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:45:08.851550 2613 hybrid_clock.cc:648] HybridClock initialized: now 1790257508851537 us; error 27 us; skew 500 ppm
Sep 24 13:45:08 dist-test-slave-tfm9 krb5kdc[2600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257508, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:45:08.854060 2613 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:45:08.854561 2613 webserver.cc:460] Webserver started at http://127.19.53.254:34109/ using document root <none> and password file <none>
I20260924 13:45:08.854815 2613 fs_manager.cc:359] Metadata directory not provided
I20260924 13:45:08.854885 2613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:45:08.855033 2613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:45:08.856086 2613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "e94886c9b8574815a47a3613ed7a4918"
format_stamp: "Formatted at 2026-09-24 13:45:08 on dist-test-slave-tfm9"
server_key: "ec6615a99739cd92f70d4608aae250a9"
server_key_iv: "c83d6869399f8c2899b96439c29b6c00"
server_key_version: "encryptionkey@0"
I20260924 13:45:08.856448 2613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "e94886c9b8574815a47a3613ed7a4918"
format_stamp: "Formatted at 2026-09-24 13:45:08 on dist-test-slave-tfm9"
server_key: "ec6615a99739cd92f70d4608aae250a9"
server_key_iv: "c83d6869399f8c2899b96439c29b6c00"
server_key_version: "encryptionkey@0"
I20260924 13:45:08.857801 2613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:45:08.858598 2628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:08.858798 2613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:45:08.858901 2613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "e94886c9b8574815a47a3613ed7a4918"
format_stamp: "Formatted at 2026-09-24 13:45:08 on dist-test-slave-tfm9"
server_key: "ec6615a99739cd92f70d4608aae250a9"
server_key_iv: "c83d6869399f8c2899b96439c29b6c00"
server_key_version: "encryptionkey@0"
I20260924 13:45:08.858983 2613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:45:08.871093 2613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:45:08.873093 2613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:45:08.873255 2613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:45:08.877728 2613 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:38623
I20260924 13:45:08.877775 2682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:38623 every 8 connection(s)
I20260924 13:45:08.878224 2613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:45:08.879451 2683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:45:08.883424 2683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Bootstrap starting.
I20260924 13:45:08.884208 2683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Neither blocks nor log segments found. Creating new log.
I20260924 13:45:08.884583 2683 log.cc:824] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Log is configured to *not* fsync() on all Append() calls
I20260924 13:45:08.885304 2683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: No bootstrap required, opened a new log
I20260924 13:45:08.885649 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 2613
I20260924 13:45:08.885778 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:45:08.886023 19671 external_mini_cluster.cc:1428] Setting key c64c3f83bd13e7b8dd276c2280c87a83
I20260924 13:45:08.888022 2683 raft_consensus.cc:348] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } }
I20260924 13:45:08.888198 2683 raft_consensus.cc:374] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:45:08.888223 2683 raft_consensus.cc:729] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94886c9b8574815a47a3613ed7a4918, State: Initialized, Role: FOLLOWER
I20260924 13:45:08.888381 2683 consensus_queue.cc:260] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } }
I20260924 13:45:08.888513 2683 raft_consensus.cc:388] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:45:08.888547 2683 raft_consensus.cc:482] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:45:08.888660 2683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:45:08.889503 2683 raft_consensus.cc:504] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } }
I20260924 13:45:08.889631 2683 leader_election.cc:302] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e94886c9b8574815a47a3613ed7a4918; no voters:
I20260924 13:45:08.889796 2683 leader_election.cc:288] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:45:08.889878 2688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:45:08.890043 2688 raft_consensus.cc:686] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 1 LEADER]: Becoming Leader. State: Replica: e94886c9b8574815a47a3613ed7a4918, State: Running, Role: LEADER
I20260924 13:45:08.890070 2683 sys_catalog.cc:564] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:45:08.891520 2688 consensus_queue.cc:237] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } }
I20260924 13:45:08.893399 2690 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e94886c9b8574815a47a3613ed7a4918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } } }
I20260924 13:45:08.893532 2690 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: This master's current role is: LEADER
I20260924 13:45:08.893399 2689 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e94886c9b8574815a47a3613ed7a4918. Latest consensus state: current_term: 1 leader_uuid: "e94886c9b8574815a47a3613ed7a4918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } } }
Sep 24 13:45:08 dist-test-slave-tfm9 krb5kdc[2600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:45:08.893761 2689 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: This master's current role is: LEADER
W20260924 13:45:08.894240 2703 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:45:08.895539 2703 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:45:08.895927 2704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:45:08.896136 2704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:45:08.898417 2704 catalog_manager.cc:1347] Generated new cluster ID: b10eea66bf844ac588ec3917df4bb91b
I20260924 13:45:08.898474 2704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:45:08.899658 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:08.886566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.19.53.254:38623)
0924 13:45:08.886753 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:08.886763 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:08.886799 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:45:08.887534 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:08.887545 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:08.887615 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:08.887783 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:45:08.888445 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:08.890443 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:08.891416 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:08.892045 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:08.892201 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:08.895143 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:45:08.895148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:45:08.895148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:45:08.895161 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:45:08.897396 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:08.897830 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:08.897833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:08.897833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:08.897995 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:08.898259 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:08.898260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:08.898260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:08.898621 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:45:08.898761 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:08.899298 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 13:45:08.899435 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260924 13:45:08.922925 2704 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:45:08.923524 2704 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:45:08.928712 2704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Generated new TSK 0
I20260924 13:45:08.928889 2704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:45:08.960070 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38623
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:45:09.040400 2709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:45:09.040594 2709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:45:09.040618 2709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:45:09.041808 2709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:45:09.041847 2709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:45:09.041873 2709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:45:09.043339 2709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:45:09.043545 2709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:45:09.043715 2709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:45:09.048738 2714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:09.048758 2717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:09.048874 2709 server_base.cc:1034] running on GCE node
W20260924 13:45:09.048753 2715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:09.049198 2709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:45:09.049443 2709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:45:09.050606 2709 hybrid_clock.cc:648] HybridClock initialized: now 1790257509050594 us; error 47 us; skew 500 ppm
Sep 24 13:45:09 dist-test-slave-tfm9 krb5kdc[2600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257509, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:45:09.063731 2709 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:45:09.064136 2709 webserver.cc:460] Webserver started at http://127.19.53.193:39601/ using document root <none> and password file <none>
I20260924 13:45:09.064338 2709 fs_manager.cc:359] Metadata directory not provided
I20260924 13:45:09.064381 2709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:45:09.064476 2709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:45:09.065477 2709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "9b0553c5fa53410880e5e1232348dc2c"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "b917a2dc734fa7924c8a1b81ab8b2e3a"
server_key_iv: "7b004f3beb074ecaf6901306b33bcff8"
server_key_version: "encryptionkey@0"
I20260924 13:45:09.065788 2709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "9b0553c5fa53410880e5e1232348dc2c"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "b917a2dc734fa7924c8a1b81ab8b2e3a"
server_key_iv: "7b004f3beb074ecaf6901306b33bcff8"
server_key_version: "encryptionkey@0"
I20260924 13:45:09.067216 2709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:45:09.067978 2724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:09.068185 2709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:45:09.068296 2709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "9b0553c5fa53410880e5e1232348dc2c"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "b917a2dc734fa7924c8a1b81ab8b2e3a"
server_key_iv: "7b004f3beb074ecaf6901306b33bcff8"
server_key_version: "encryptionkey@0"
I20260924 13:45:09.068372 2709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:45:09.078258 2709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:45:09.080471 2709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:45:09.080631 2709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:45:09.080852 2709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:45:09.081147 2709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:45:09.081203 2709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:09.081243 2709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:45:09.081272 2709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:09.086978 2709 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:39633
I20260924 13:45:09.087023 2839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:39633 every 8 connection(s)
I20260924 13:45:09.087339 2709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:45:09 dist-test-slave-tfm9 krb5kdc[2600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257509, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:45:09.094949 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 2709
I20260924 13:45:09.095094 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:45:09.095338 19671 external_mini_cluster.cc:1428] Setting key 933d88f659658db866a031ab81a10410
I20260924 13:45:09.097168 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.088031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:41605 (local address 127.19.53.254:38623)
0924 13:45:09.088094 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:09.088097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:09.088310 (+ 213us) server_negotiation.cc:407] Connection header received
0924 13:45:09.088773 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:09.088775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:09.088784 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:09.088846 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:45:09.089745 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.090354 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.091535 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.091781 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.091844 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.093947 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:45:09.093948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:45:09.093949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:45:09.093954 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:45:09.095575 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:09.096049 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:09.096050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:09.096051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:09.096138 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:09.096445 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:09.096446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:09.096446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:09.096569 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:45:09.096656 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:09.096964 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 13:45:09.097058 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:45:09.097389 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.088017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:41605)
0924 13:45:09.088177 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:09.088217 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:09.088691 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:09.088900 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:09.088903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:09.089455 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:45:09.089647 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.089651 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.090433 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.090433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.091463 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.091465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.091853 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.091854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.091962 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.092499 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:45:09.092507 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:45:09.093845 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:45:09.095688 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:09.095696 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:09.095699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:09.095949 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:09.096242 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:09.096243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:09.096244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:09.096367 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:09.096670 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:45:09.096678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:45:09.096815 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 13:45:09.096941 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 13:45:09.097105 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260924 13:45:09.097929 2840 heartbeater.cc:347] Connected to a master server at 127.19.53.254:38623
I20260924 13:45:09.098026 2840 heartbeater.cc:464] Registering TS with master...
I20260924 13:45:09.098172 2840 heartbeater.cc:511] Master 127.19.53.254:38623 requested a full tablet report, sending...
I20260924 13:45:09.098678 2646 ts_manager.cc:194] Registered new tserver with Master: 9b0553c5fa53410880e5e1232348dc2c (127.19.53.193:39633)
I20260924 13:45:09.099320 2646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:41605
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:45:09.120079 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38623
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:45:09.203131 2845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:45:09.203327 2845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:45:09.203349 2845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:45:09.204474 2845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:45:09.204515 2845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:45:09.204540 2845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:45:09.206014 2845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:45:09.206251 2845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:45:09.206481 2845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:45:09.211309 2850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:09.211462 2845 server_base.cc:1034] running on GCE node
W20260924 13:45:09.211324 2851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:09.211306 2853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:09.211815 2845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:45:09.211995 2845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:45:09.213124 2845 hybrid_clock.cc:648] HybridClock initialized: now 1790257509213115 us; error 30 us; skew 500 ppm
Sep 24 13:45:09 dist-test-slave-tfm9 krb5kdc[2600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257509, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:45:09.215618 2845 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:45:09.216033 2845 webserver.cc:460] Webserver started at http://127.19.53.194:36377/ using document root <none> and password file <none>
I20260924 13:45:09.216239 2845 fs_manager.cc:359] Metadata directory not provided
I20260924 13:45:09.216279 2845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:45:09.216363 2845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:45:09.217298 2845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "f1aecb9292344cf4b8bb47a2671db0cb"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "513063f9ed0c52412e64a23dc93efb29"
server_key_iv: "760bf467403fdf7bc0fa56029f347b21"
server_key_version: "encryptionkey@0"
I20260924 13:45:09.217588 2845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "f1aecb9292344cf4b8bb47a2671db0cb"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "513063f9ed0c52412e64a23dc93efb29"
server_key_iv: "760bf467403fdf7bc0fa56029f347b21"
server_key_version: "encryptionkey@0"
I20260924 13:45:09.218844 2845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:45:09.219565 2861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:09.219771 2845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:45:09.219861 2845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "f1aecb9292344cf4b8bb47a2671db0cb"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "513063f9ed0c52412e64a23dc93efb29"
server_key_iv: "760bf467403fdf7bc0fa56029f347b21"
server_key_version: "encryptionkey@0"
I20260924 13:45:09.219923 2845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:45:09.239441 2845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:45:09.241202 2845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:45:09.241361 2845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:45:09.241576 2845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:45:09.241858 2845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:45:09.241890 2845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:09.241955 2845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:45:09.241982 2845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:09.247720 2845 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:36563
I20260924 13:45:09.247774 2975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:36563 every 8 connection(s)
I20260924 13:45:09.248076 2845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:45:09 dist-test-slave-tfm9 krb5kdc[2600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257509, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:45:09.254527 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 2845
I20260924 13:45:09.254643 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:45:09.254858 19671 external_mini_cluster.cc:1428] Setting key 7b1a49d3c726786b044e8817e314d103
I20260924 13:45:09.257457 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.248797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:57221 (local address 127.19.53.254:38623)
0924 13:45:09.248871 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:09.248873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:09.249034 (+ 161us) server_negotiation.cc:407] Connection header received
0924 13:45:09.249557 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:09.249559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:09.249572 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:09.249623 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:45:09.250598 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.251106 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.252092 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.252291 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.252399 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.254375 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:45:09.254377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:45:09.254377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:45:09.254385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:45:09.255957 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:09.256353 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:09.256354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:09.256355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:09.256415 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:09.256725 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:09.256726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:09.256727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:09.256886 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:45:09.256949 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:09.257324 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 13:45:09.257393 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 13:45:09.257722 2978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.248750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:57221)
0924 13:45:09.248915 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:09.248959 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:09.249445 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:09.249688 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:09.249692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:09.250317 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:45:09.250498 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.250503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.251204 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.251205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.251990 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.251991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.252407 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.252409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.252533 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.252958 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:45:09.252966 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:45:09.254262 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:45:09.256040 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:09.256047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:09.256049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:09.256269 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:09.256486 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:09.256487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:09.256487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:09.256635 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:09.256959 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:45:09.256965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:45:09.257177 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 13:45:09.257299 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 13:45:09.257469 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260924 13:45:09.258250 2976 heartbeater.cc:347] Connected to a master server at 127.19.53.254:38623
I20260924 13:45:09.258347 2976 heartbeater.cc:464] Registering TS with master...
I20260924 13:45:09.258525 2976 heartbeater.cc:511] Master 127.19.53.254:38623 requested a full tablet report, sending...
I20260924 13:45:09.258845 2646 ts_manager.cc:194] Registered new tserver with Master: f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563)
I20260924 13:45:09.259246 2646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:57221
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:45:09.279459 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38623
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:45:09.357976 2981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:45:09.358217 2981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:45:09.358260 2981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:45:09.359485 2981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:45:09.359555 2981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:45:09.359599 2981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:45:09.361025 2981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:45:09.361394 2981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:45:09.361629 2981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:45:09.366464 2986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:09.366487 2989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:09.366460 2987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:09.366494 2981 server_base.cc:1034] running on GCE node
I20260924 13:45:09.366959 2981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:45:09.367195 2981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:45:09.368356 2981 hybrid_clock.cc:648] HybridClock initialized: now 1790257509368340 us; error 43 us; skew 500 ppm
Sep 24 13:45:09 dist-test-slave-tfm9 krb5kdc[2600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257509, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:45:09.371018 2981 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:45:09.371469 2981 webserver.cc:460] Webserver started at http://127.19.53.195:40867/ using document root <none> and password file <none>
I20260924 13:45:09.371644 2981 fs_manager.cc:359] Metadata directory not provided
I20260924 13:45:09.371680 2981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:45:09.371763 2981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:45:09.372691 2981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "48e03d2048ab4d329f0b715d8e1a07e4"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "c481ed5f8a1c699999feff482ed1f8da"
server_key_iv: "b9dc0681dee9225d9a04a975211b814b"
server_key_version: "encryptionkey@0"
I20260924 13:45:09.372977 2981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "48e03d2048ab4d329f0b715d8e1a07e4"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "c481ed5f8a1c699999feff482ed1f8da"
server_key_iv: "b9dc0681dee9225d9a04a975211b814b"
server_key_version: "encryptionkey@0"
I20260924 13:45:09.374372 2981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:45:09.375144 2996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:09.375384 2981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:45:09.375507 2981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "48e03d2048ab4d329f0b715d8e1a07e4"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "c481ed5f8a1c699999feff482ed1f8da"
server_key_iv: "b9dc0681dee9225d9a04a975211b814b"
server_key_version: "encryptionkey@0"
I20260924 13:45:09.375584 2981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:45:09.389609 2981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:45:09.391693 2981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:45:09.391845 2981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:45:09.392051 2981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:45:09.392338 2981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:45:09.392391 2981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:09.392431 2981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:45:09.392459 2981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:09.398154 2981 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:37809
I20260924 13:45:09.398214 3110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:37809 every 8 connection(s)
I20260924 13:45:09.398527 2981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:45:09.403450 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 2981
I20260924 13:45:09.403550 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:45:09.403714 19671 external_mini_cluster.cc:1428] Setting key eeabc775a03643b3b3d4d56204fbd2f0
Sep 24 13:45:09 dist-test-slave-tfm9 krb5kdc[2600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257509, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:45:09.407663 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.399245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:35859 (local address 127.19.53.254:38623)
0924 13:45:09.399315 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:09.399317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:09.399595 (+ 278us) server_negotiation.cc:407] Connection header received
0924 13:45:09.400152 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:09.400154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:09.400168 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:09.400225 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:45:09.401210 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.401771 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.402849 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.403089 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.403180 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.405339 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:45:09.405340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:45:09.405341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:45:09.405347 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:45:09.406512 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:09.406901 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:09.406902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:09.406902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:09.406952 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:09.407132 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:09.407133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:09.407133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:09.407230 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:45:09.407287 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:09.407527 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 13:45:09.407583 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 13:45:09.407917 3113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.399258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:35859)
0924 13:45:09.399458 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:09.399493 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:09.400010 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:09.400304 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:09.400307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:09.400900 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:45:09.401099 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.401104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.401871 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.401872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.402749 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.402752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.403191 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.403192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.403335 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.403869 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:45:09.403886 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:45:09.405257 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:45:09.406605 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:09.406611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:09.406612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:09.406833 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:09.406996 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:09.406996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:09.406996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:09.407080 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:09.407308 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:45:09.407312 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:45:09.407421 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 13:45:09.407505 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 13:45:09.407649 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":106,"threads_started":1}
I20260924 13:45:09.408528 3111 heartbeater.cc:347] Connected to a master server at 127.19.53.254:38623
I20260924 13:45:09.408641 3111 heartbeater.cc:464] Registering TS with master...
I20260924 13:45:09.408808 3111 heartbeater.cc:511] Master 127.19.53.254:38623 requested a full tablet report, sending...
I20260924 13:45:09.409178 2646 ts_manager.cc:194] Registered new tserver with Master: 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809)
I20260924 13:45:09.409543 2646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:35859
I20260924 13:45:09.415298 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:45:09.421972 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.416552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52046 (local address 127.19.53.254:38623)
0924 13:45:09.416616 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:09.416618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:09.416731 (+ 113us) server_negotiation.cc:407] Connection header received
0924 13:45:09.416763 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:09.416764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:09.416772 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:09.416835 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:45:09.417327 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.417774 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.418491 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.418663 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.418714 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.419821 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:45:09.419822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:45:09.419822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:45:09.419827 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:45:09.420962 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:09.421282 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:09.421282 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:09.421283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:09.421330 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:09.421484 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:09.421484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:09.421484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:09.421606 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:45:09.421686 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:09.421787 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 13:45:09.421842 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 13:45:09.423638 2646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:45:09.425186 2646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:45:09.431197 3122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.427104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39633 (local address 127.0.0.1:37002)
0924 13:45:09.427604 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:09.427615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:09.427667 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:09.427901 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:09.427902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:09.427903 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:45:09.427937 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.427938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.429061 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.429061 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.430329 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.430332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.430988 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.430988 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.431040 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.431051 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 13:45:09.431107 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 13:45:09.431143 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":435,"thread_start_us":49,"threads_started":1}
I20260924 13:45:09.431273 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.427203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.19.53.193:39633)
0924 13:45:09.427425 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:09.427428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:09.427730 (+ 302us) server_negotiation.cc:407] Connection header received
0924 13:45:09.427779 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:09.427783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:09.427816 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:09.427895 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:45:09.428080 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.428945 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.430426 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.430916 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.430976 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.431012 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:09.431148 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 13:45:09.431187 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
I20260924 13:45:09.432010 3119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.427102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:37809 (local address 127.0.0.1:57588)
0924 13:45:09.427222 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:09.427259 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:09.427328 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:09.427521 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:09.427525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:09.427527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:45:09.427587 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.427589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.428888 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.428889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.430329 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.430332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.431764 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.431765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.431841 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.431845 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:45:09.431901 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 13:45:09.431945 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":59,"thread_start_us":51,"threads_started":1}
I20260924 13:45:09.432045 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.427159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.19.53.195:37809)
0924 13:45:09.427351 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:09.427356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:09.427371 (+ 15us) server_negotiation.cc:407] Connection header received
0924 13:45:09.427409 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:09.427411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:09.427427 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:09.427502 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:45:09.427692 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.428793 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.431041 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.431659 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.431757 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.431822 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:09.431929 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 13:45:09.431972 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":85,"threads_started":1}
I20260924 13:45:09.432117 2773 tablet_service.cc:1467] Processing CreateTablet for tablet 4b04239ee7ba4729bdf30464400dd7b3 (DEFAULT_TABLE table=test-table [id=589f21d12a684ebb94751b67e79b17ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:45:09.432479 2773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b04239ee7ba4729bdf30464400dd7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:45:09.432955 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.427336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:36563 (local address 127.0.0.1:33484)
0924 13:45:09.427586 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:09.427596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:09.427635 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:09.428241 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:09.428242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:09.428245 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:45:09.428289 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.428290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.429420 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.429421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.430338 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.430341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.432720 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.432721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.432782 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.432787 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:45:09.432846 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 13:45:09.432883 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":49,"threads_started":1}
I20260924 13:45:09.432849 3045 tablet_service.cc:1467] Processing CreateTablet for tablet 4b04239ee7ba4729bdf30464400dd7b3 (DEFAULT_TABLE table=test-table [id=589f21d12a684ebb94751b67e79b17ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:45:09.432971 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.427517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.19.53.194:36563)
0924 13:45:09.428024 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:09.428029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:09.428037 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:45:09.428072 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:09.428074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:09.428088 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:09.428170 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:45:09.428362 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.429295 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.431858 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.432634 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.432708 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.432757 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:09.432855 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 13:45:09.432894 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":95,"threads_started":1}
I20260924 13:45:09.433169 3045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b04239ee7ba4729bdf30464400dd7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:45:09.433593 2909 tablet_service.cc:1467] Processing CreateTablet for tablet 4b04239ee7ba4729bdf30464400dd7b3 (DEFAULT_TABLE table=test-table [id=589f21d12a684ebb94751b67e79b17ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:45:09.433893 2909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b04239ee7ba4729bdf30464400dd7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:45:09.435753 3125 tablet_bootstrap.cc:492] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4: Bootstrap starting.
I20260924 13:45:09.436354 3125 tablet_bootstrap.cc:654] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4: Neither blocks nor log segments found. Creating new log.
I20260924 13:45:09.436383 3126 tablet_bootstrap.cc:492] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Bootstrap starting.
I20260924 13:45:09.436868 3125 log.cc:824] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4: Log is configured to *not* fsync() on all Append() calls
I20260924 13:45:09.437018 3126 tablet_bootstrap.cc:654] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Neither blocks nor log segments found. Creating new log.
I20260924 13:45:09.437259 3127 tablet_bootstrap.cc:492] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb: Bootstrap starting.
I20260924 13:45:09.437572 3126 log.cc:824] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Log is configured to *not* fsync() on all Append() calls
I20260924 13:45:09.437810 3127 tablet_bootstrap.cc:654] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb: Neither blocks nor log segments found. Creating new log.
I20260924 13:45:09.438297 3126 tablet_bootstrap.cc:492] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: No bootstrap required, opened a new log
I20260924 13:45:09.438305 3127 log.cc:824] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb: Log is configured to *not* fsync() on all Append() calls
I20260924 13:45:09.438366 3126 ts_tablet_manager.cc:1397] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:45:09.439088 3127 tablet_bootstrap.cc:492] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb: No bootstrap required, opened a new log
I20260924 13:45:09.439100 3125 tablet_bootstrap.cc:492] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4: No bootstrap required, opened a new log
I20260924 13:45:09.439160 3127 ts_tablet_manager.cc:1397] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 13:45:09.439160 3125 ts_tablet_manager.cc:1397] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 13:45:09.439945 3126 raft_consensus.cc:348] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } } peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } }
I20260924 13:45:09.440104 3126 raft_consensus.cc:374] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:45:09.440148 3126 raft_consensus.cc:729] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0553c5fa53410880e5e1232348dc2c, State: Initialized, Role: FOLLOWER
I20260924 13:45:09.440292 3126 consensus_queue.cc:260] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } } peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } }
I20260924 13:45:09.440557 3126 ts_tablet_manager.cc:1428] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:45:09.440707 2840 heartbeater.cc:503] Master 127.19.53.254:38623 was elected leader, sending a full tablet report...
I20260924 13:45:09.441073 3125 raft_consensus.cc:348] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } } peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } }
I20260924 13:45:09.441073 3127 raft_consensus.cc:348] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } } peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } }
I20260924 13:45:09.441246 3127 raft_consensus.cc:374] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:45:09.441246 3125 raft_consensus.cc:374] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:45:09.441282 3127 raft_consensus.cc:729] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1aecb9292344cf4b8bb47a2671db0cb, State: Initialized, Role: FOLLOWER
I20260924 13:45:09.441282 3125 raft_consensus.cc:729] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e03d2048ab4d329f0b715d8e1a07e4, State: Initialized, Role: FOLLOWER
I20260924 13:45:09.441390 3127 consensus_queue.cc:260] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } } peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } }
I20260924 13:45:09.441390 3125 consensus_queue.cc:260] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } } peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } }
I20260924 13:45:09.441648 3127 ts_tablet_manager.cc:1428] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 13:45:09.441648 3125 ts_tablet_manager.cc:1428] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 13:45:09.441687 2976 heartbeater.cc:503] Master 127.19.53.254:38623 was elected leader, sending a full tablet report...
I20260924 13:45:09.441977 3111 heartbeater.cc:503] Master 127.19.53.254:38623 was elected leader, sending a full tablet report...
I20260924 13:45:09.466560 3131 raft_consensus.cc:482] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:45:09.466672 3131 raft_consensus.cc:504] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } } peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } }
I20260924 13:45:09.467149 3131 leader_election.cc:288] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809), f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563)
I20260924 13:45:09.470022 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.467468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:52951 (local address 127.19.53.195:37809)
0924 13:45:09.467528 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:09.467530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:09.467539 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:45:09.467564 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:09.467566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:09.467575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:09.467644 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:45:09.467813 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.468285 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.469213 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.469665 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.469731 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.469742 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:09.469904 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 13:45:09.469960 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 13:45:09.470021 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.467350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:37809 (local address 127.19.53.193:52951)
0924 13:45:09.467441 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:09.467454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:09.467507 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:09.467659 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:09.467660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:09.467662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:45:09.467737 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.467739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.468369 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.468371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.469094 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.469095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.469740 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.469741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.469825 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.469828 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:45:09.469897 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 13:45:09.469952 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 13:45:09.470379 3065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b04239ee7ba4729bdf30464400dd7b3" candidate_uuid: "9b0553c5fa53410880e5e1232348dc2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" is_pre_election: true
I20260924 13:45:09.470564 3065 raft_consensus.cc:2454] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b0553c5fa53410880e5e1232348dc2c in term 0.
I20260924 13:45:09.470592 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.467519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:36563 (local address 127.19.53.193:40693)
0924 13:45:09.467697 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:09.467707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:09.467773 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:09.467910 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:09.467917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:09.467919 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:45:09.467993 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.468000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.468691 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.468692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.469512 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:09.469515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.470314 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.470315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:09.470397 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.470401 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:45:09.470469 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 13:45:09.470524 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":110,"threads_started":1}
I20260924 13:45:09.470610 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.467637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:40693 (local address 127.19.53.194:36563)
0924 13:45:09.467689 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:09.467691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:09.467783 (+ 92us) server_negotiation.cc:407] Connection header received
0924 13:45:09.467835 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:09.467836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:09.467846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:09.467897 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:45:09.468074 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.468584 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.469627 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.470226 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.470311 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.470320 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:09.470476 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 13:45:09.470534 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 13:45:09.470811 2725 leader_election.cc:302] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e03d2048ab4d329f0b715d8e1a07e4, 9b0553c5fa53410880e5e1232348dc2c; no voters:
I20260924 13:45:09.470925 3131 raft_consensus.cc:2781] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:45:09.470988 3131 raft_consensus.cc:482] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:45:09.470935 2930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b04239ee7ba4729bdf30464400dd7b3" candidate_uuid: "9b0553c5fa53410880e5e1232348dc2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" is_pre_election: true
I20260924 13:45:09.471015 3131 raft_consensus.cc:3037] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:45:09.471086 2930 raft_consensus.cc:2454] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b0553c5fa53410880e5e1232348dc2c in term 0.
I20260924 13:45:09.471879 3131 raft_consensus.cc:504] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } } peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } }
I20260924 13:45:09.472014 3131 leader_election.cc:288] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 1 election: Requested vote from peers 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809), f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563)
I20260924 13:45:09.472172 3065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b04239ee7ba4729bdf30464400dd7b3" candidate_uuid: "9b0553c5fa53410880e5e1232348dc2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e03d2048ab4d329f0b715d8e1a07e4"
I20260924 13:45:09.472242 2930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b04239ee7ba4729bdf30464400dd7b3" candidate_uuid: "9b0553c5fa53410880e5e1232348dc2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1aecb9292344cf4b8bb47a2671db0cb"
I20260924 13:45:09.472275 3065 raft_consensus.cc:3037] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:45:09.472306 2930 raft_consensus.cc:3037] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:45:09.473281 3065 raft_consensus.cc:2454] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0553c5fa53410880e5e1232348dc2c in term 1.
I20260924 13:45:09.473281 2930 raft_consensus.cc:2454] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0553c5fa53410880e5e1232348dc2c in term 1.
I20260924 13:45:09.473469 2725 leader_election.cc:302] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e03d2048ab4d329f0b715d8e1a07e4, 9b0553c5fa53410880e5e1232348dc2c; no voters:
I20260924 13:45:09.473575 3131 raft_consensus.cc:2781] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:45:09.473714 3131 raft_consensus.cc:686] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 1 LEADER]: Becoming Leader. State: Replica: 9b0553c5fa53410880e5e1232348dc2c, State: Running, Role: LEADER
I20260924 13:45:09.473803 3131 consensus_queue.cc:237] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } } peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } }
I20260924 13:45:09.474611 2646 catalog_manager.cc:5495] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b0553c5fa53410880e5e1232348dc2c (127.19.53.193). New cstate: current_term: 1 leader_uuid: "9b0553c5fa53410880e5e1232348dc2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } health_report { overall_health: UNKNOWN } } }
I20260924 13:45:09.488296 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.486176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37004 (local address 127.19.53.193:39633)
0924 13:45:09.486236 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:09.486237 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:09.486244 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:45:09.486288 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:09.486289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:09.486304 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:09.486359 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:45:09.486491 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.486946 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.487634 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:09.487813 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:09.487861 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:09.487992 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:45:09.488079 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:45:09.488136 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:09.488166 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 13:45:09.488182 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:45:09.490062 2930 raft_consensus.cc:1261] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb [term 1 FOLLOWER]: Refusing update from remote peer 9b0553c5fa53410880e5e1232348dc2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:45:09.490062 3065 raft_consensus.cc:1261] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4 [term 1 FOLLOWER]: Refusing update from remote peer 9b0553c5fa53410880e5e1232348dc2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:45:09.490387 3131 consensus_queue.cc:1035] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:45:09.490509 3135 consensus_queue.cc:1035] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:45:09.492966 3139 mvcc.cc:205] Tried to move back new op lower bound from 7332894758868377600 to 7332894758805295104. Current Snapshot: MvccSnapshot[applied={T|T < 7332894758868377600}]
I20260924 13:45:09.492966 3140 mvcc.cc:205] Tried to move back new op lower bound from 7332894758868377600 to 7332894758805295104. Current Snapshot: MvccSnapshot[applied={T|T < 7332894758868377600}]
I20260924 13:45:09.494971 2645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:45:09.495066 2645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:45:09.495831 2645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Sending DeleteTablet for 3 replicas of tablet 4b04239ee7ba4729bdf30464400dd7b3
I20260924 13:45:09.496167 2773 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b04239ee7ba4729bdf30464400dd7b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:45:09 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:37002
I20260924 13:45:09.496206 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 2709
I20260924 13:45:09.496732 3045 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b04239ee7ba4729bdf30464400dd7b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:45:09 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:57588
I20260924 13:45:09.496933 3149 tablet_replica.cc:330] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4: stopping tablet replica
I20260924 13:45:09.497008 3149 raft_consensus.cc:2229] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:45:09.497112 3149 raft_consensus.cc:2258] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:45:09.497161 2909 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b04239ee7ba4729bdf30464400dd7b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:45:09 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:33484
I20260924 13:45:09.497412 3150 tablet_replica.cc:330] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb: stopping tablet replica
I20260924 13:45:09.497503 3150 raft_consensus.cc:2229] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:45:09.497570 3149 ts_tablet_manager.cc:1905] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:45:09.497601 3150 raft_consensus.cc:2258] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:45:09.499157 3149 ts_tablet_manager.cc:1918] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:45:09.499250 3149 log.cc:1196] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/4b04239ee7ba4729bdf30464400dd7b3
I20260924 13:45:09.499526 3149 ts_tablet_manager.cc:1939] T 4b04239ee7ba4729bdf30464400dd7b3 P 48e03d2048ab4d329f0b715d8e1a07e4: Deleting consensus metadata
I20260924 13:45:09.499524 3150 ts_tablet_manager.cc:1905] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:45:09.499907 2629 catalog_manager.cc:4841] TS 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809): tablet 4b04239ee7ba4729bdf30464400dd7b3 (table test-table [id=589f21d12a684ebb94751b67e79b17ac]) successfully deleted
I20260924 13:45:09.501456 3150 ts_tablet_manager.cc:1918] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:45:09.501529 3150 log.cc:1196] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/4b04239ee7ba4729bdf30464400dd7b3
I20260924 13:45:09.501785 3150 ts_tablet_manager.cc:1939] T 4b04239ee7ba4729bdf30464400dd7b3 P f1aecb9292344cf4b8bb47a2671db0cb: Deleting consensus metadata
I20260924 13:45:09.502343 2629 catalog_manager.cc:4841] TS f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563): tablet 4b04239ee7ba4729bdf30464400dd7b3 (table test-table [id=589f21d12a684ebb94751b67e79b17ac]) successfully deleted
W20260924 13:45:09.502691 2632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:39633 (error 108)
I20260924 13:45:09.502946 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 2845
I20260924 13:45:09.503120 3119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:09.503002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39633 (local address 127.0.0.1:37006)
0924 13:45:09.503051 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:09.503083 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:39633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 13:45:09.503221 2632 catalog_manager.cc:4584] TS 9b0553c5fa53410880e5e1232348dc2c (127.19.53.193:39633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b04239ee7ba4729bdf30464400dd7b3: Network error: Client connection negotiation failed: client connection to 127.19.53.193:39633: connect: Connection refused (error 111)
I20260924 13:45:09.508483 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 2981
I20260924 13:45:09.512782 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 2613
I20260924 13:45:29.517969 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:38623
--webserver_interface=127.19.53.254
--webserver_port=34109
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:45:29.602187 3152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:45:29.602430 3152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:45:29.602479 3152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:45:29.603701 3152 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 13:45:29.603773 3152 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 13:45:29.603798 3152 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 13:45:29.603829 3152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:45:29.603860 3152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:45:29.603892 3152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:45:29.603922 3152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:45:29.603955 3152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:45:29.605479 3152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:38623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:45:29.605759 3152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:45:29.606066 3152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:45:29.615048 3159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:29.615043 3158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:29.615031 3161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:29.615072 3152 server_base.cc:1034] running on GCE node
I20260924 13:45:29.615518 3152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:45:29.615872 3152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:45:29.617048 3152 hybrid_clock.cc:648] HybridClock initialized: now 1790257529617037 us; error 48 us; skew 500 ppm
Sep 24 13:45:29 dist-test-slave-tfm9 krb5kdc[2600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257529, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:45:29.619609 3152 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:45:29.620078 3152 webserver.cc:460] Webserver started at http://127.19.53.254:34109/ using document root <none> and password file <none>
I20260924 13:45:29.620272 3152 fs_manager.cc:359] Metadata directory not provided
I20260924 13:45:29.620328 3152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:45:29.621647 3152 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:45:29.622547 3168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:29.622749 3152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:45:29.622836 3152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "e94886c9b8574815a47a3613ed7a4918"
format_stamp: "Formatted at 2026-09-24 13:45:08 on dist-test-slave-tfm9"
server_key: "ec6615a99739cd92f70d4608aae250a9"
server_key_iv: "c83d6869399f8c2899b96439c29b6c00"
server_key_version: "encryptionkey@0"
I20260924 13:45:29.623229 3152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:45:29.643971 3152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:45:29.646353 3152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:45:29.646466 3152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:45:29.650255 3152 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:38623
I20260924 13:45:29.650305 3220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:38623 every 8 connection(s)
I20260924 13:45:29.650689 3152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:45:29.652253 3221 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:45:29.652930 3221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Bootstrap starting.
I20260924 13:45:29.653764 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 3152
I20260924 13:45:29.654070 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:39633
--local_ip_for_outbound_sockets=127.19.53.193
--tserver_master_addrs=127.19.53.254:38623
--webserver_port=39601
--webserver_interface=127.19.53.193
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:45:29.661878 3221 log.cc:824] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Log is configured to *not* fsync() on all Append() calls
I20260924 13:45:29.663789 3221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:45:29.664098 3221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Bootstrap complete.
I20260924 13:45:29.666510 3221 raft_consensus.cc:348] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } }
I20260924 13:45:29.666756 3221 raft_consensus.cc:729] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94886c9b8574815a47a3613ed7a4918, State: Initialized, Role: FOLLOWER
I20260924 13:45:29.666972 3221 consensus_queue.cc:260] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } }
I20260924 13:45:29.667091 3221 raft_consensus.cc:388] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:45:29.667150 3221 raft_consensus.cc:482] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:45:29.667227 3221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:45:29.668198 3221 raft_consensus.cc:504] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } }
I20260924 13:45:29.668321 3221 leader_election.cc:302] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e94886c9b8574815a47a3613ed7a4918; no voters:
I20260924 13:45:29.668499 3221 leader_election.cc:288] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 13:45:29.668576 3225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:45:29.668758 3221 sys_catalog.cc:564] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:45:29.668812 3225 raft_consensus.cc:686] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 2 LEADER]: Becoming Leader. State: Replica: e94886c9b8574815a47a3613ed7a4918, State: Running, Role: LEADER
I20260924 13:45:29.668999 3225 consensus_queue.cc:237] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } }
I20260924 13:45:29.669344 3226 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e94886c9b8574815a47a3613ed7a4918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } } }
I20260924 13:45:29.669416 3226 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: This master's current role is: LEADER
I20260924 13:45:29.669548 3227 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e94886c9b8574815a47a3613ed7a4918. Latest consensus state: current_term: 2 leader_uuid: "e94886c9b8574815a47a3613ed7a4918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } } }
I20260924 13:45:29.669601 3227 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: This master's current role is: LEADER
I20260924 13:45:29.670595 3241 catalog_manager.cc:1259] Loaded cluster ID: b10eea66bf844ac588ec3917df4bb91b
I20260924 13:45:29.670646 3241 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: loading cluster ID for follower catalog manager: success
I20260924 13:45:29.671789 3241 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: acquiring CA information for follower catalog manager: success
I20260924 13:45:29.672197 3241 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:45:29.672269 3234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:45:29.672668 3234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:45:29.672744 3234 catalog_manager.cc:1259] Loaded cluster ID: b10eea66bf844ac588ec3917df4bb91b
I20260924 13:45:29.672767 3234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:45:29.672916 3234 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:45:29.672981 3234 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Loaded TSK: 0
I20260924 13:45:29.700626 3234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Generated new TSK 1
I20260924 13:45:29.700747 3234 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:45:29.738865 3223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:45:29.739084 3223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:45:29.739121 3223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:45:29.740216 3223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:45:29.740288 3223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:45:29.740330 3223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:45:29.741789 3223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:39633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=39601
--tserver_master_addrs=127.19.53.254:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:45:29.742087 3223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:45:29.742319 3223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:45:29.748202 3249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:29.748471 3248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:29.748610 3223 server_base.cc:1034] running on GCE node
W20260924 13:45:29.748772 3251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:29.748996 3223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:45:29.749275 3223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:45:29.750427 3223 hybrid_clock.cc:648] HybridClock initialized: now 1790257529750415 us; error 28 us; skew 500 ppm
Sep 24 13:45:29 dist-test-slave-tfm9 krb5kdc[2600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257529, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:45:29.753046 3223 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:45:29.753448 3223 webserver.cc:460] Webserver started at http://127.19.53.193:39601/ using document root <none> and password file <none>
I20260924 13:45:29.753701 3223 fs_manager.cc:359] Metadata directory not provided
I20260924 13:45:29.753786 3223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:45:29.755260 3223 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:45:29.756028 3258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:29.756263 3223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:45:29.756362 3223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "9b0553c5fa53410880e5e1232348dc2c"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "b917a2dc734fa7924c8a1b81ab8b2e3a"
server_key_iv: "7b004f3beb074ecaf6901306b33bcff8"
server_key_version: "encryptionkey@0"
I20260924 13:45:29.756727 3223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:45:29.766359 3223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:45:29.768532 3223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:45:29.768687 3223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:45:29.768903 3223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:45:29.769488 3265 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 13:45:29.771428 3223 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 13:45:29.771490 3223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 13:45:29.771534 3223 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 13:45:29.772135 3223 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 13:45:29.772192 3223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 13:45:29.772256 3265 tablet_bootstrap.cc:492] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Bootstrap starting.
I20260924 13:45:29.779407 3223 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:39633
I20260924 13:45:29.779760 3223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:45:29.782142 3373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:39633 every 8 connection(s)
I20260924 13:45:29.783784 3265 log.cc:824] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Log is configured to *not* fsync() on all Append() calls
I20260924 13:45:29.786593 3265 tablet_bootstrap.cc:492] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:45:29.786906 3265 tablet_bootstrap.cc:492] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Bootstrap complete.
I20260924 13:45:29.788115 3265 ts_tablet_manager.cc:1397] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260924 13:45:29.789542 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 3223
I20260924 13:45:29.789855 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:36563
--local_ip_for_outbound_sockets=127.19.53.194
--tserver_master_addrs=127.19.53.254:38623
--webserver_port=36377
--webserver_interface=127.19.53.194
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:45:29 dist-test-slave-tfm9 krb5kdc[2600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257529, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:45:29.791539 3265 raft_consensus.cc:348] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } } peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } }
I20260924 13:45:29.791961 3265 raft_consensus.cc:729] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0553c5fa53410880e5e1232348dc2c, State: Initialized, Role: FOLLOWER
I20260924 13:45:29.792186 3265 consensus_queue.cc:260] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } } peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } }
I20260924 13:45:29.792534 3265 ts_tablet_manager.cc:1428] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260924 13:45:29.794663 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:29.780467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:37623)
0924 13:45:29.780655 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:29.780696 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:29.781155 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:29.784534 (+ 3379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:29.784538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:29.784823 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:45:29.785002 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:29.785007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:29.786544 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:29.786545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:29.787487 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:29.787489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:29.788112 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:29.788113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:29.788255 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:29.788815 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:45:29.788823 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:45:29.790330 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:45:29.792725 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:29.792732 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:29.792734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:29.793031 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:29.793329 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:29.793330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:29.793330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:29.793464 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:29.793949 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:45:29.793956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:45:29.794129 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 13:45:29.794228 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 13:45:29.794412 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":116,"threads_started":1}
I20260924 13:45:29.795346 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:29.780544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:37623 (local address 127.19.53.254:38623)
0924 13:45:29.784072 (+ 3528us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:29.784082 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:29.784129 (+ 47us) server_negotiation.cc:407] Connection header received
0924 13:45:29.784267 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:29.784275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:29.784337 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:29.784504 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:45:29.785146 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:29.786421 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:29.787586 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:29.788003 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:29.788136 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:29.790508 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:45:29.790512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:45:29.790512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:45:29.790523 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:45:29.792627 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:29.793140 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:29.793143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:29.793143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:29.793258 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:29.793536 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:29.793537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:29.793537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:29.793828 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:45:29.794830 (+ 1002us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:29.794922 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 13:45:29.795084 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3470,"thread_start_us":95,"threads_started":1}
I20260924 13:45:29.796103 3374 heartbeater.cc:347] Connected to a master server at 127.19.53.254:38623
I20260924 13:45:29.796200 3374 heartbeater.cc:464] Registering TS with master...
I20260924 13:45:29.796373 3374 heartbeater.cc:511] Master 127.19.53.254:38623 requested a full tablet report, sending...
I20260924 13:45:29.797227 3185 ts_manager.cc:194] Registered new tserver with Master: 9b0553c5fa53410880e5e1232348dc2c (127.19.53.193:39633)
I20260924 13:45:29.798607 3185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:37623
I20260924 13:45:29.799723 3374 heartbeater.cc:503] Master 127.19.53.254:38623 was elected leader, sending a full tablet report...
Sep 24 13:45:29 dist-test-slave-tfm9 krb5kdc[2600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257529, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for kudu/127.19.53.193@KRBTEST.COM
I20260924 13:45:29.808256 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:29.798172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.19.53.193:39633)
0924 13:45:29.798549 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:29.798553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:29.798574 (+ 21us) server_negotiation.cc:407] Connection header received
0924 13:45:29.798606 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:29.798608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:29.798621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:29.798686 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:45:29.799548 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:29.800871 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:29.802165 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:29.802604 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:29.802684 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:29.805103 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:45:29.805110 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:45:29.805111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:45:29.805117 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:45:29.806896 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:29.807360 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:29.807363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:29.807364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:29.807427 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:29.807632 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:29.807633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:29.807633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:29.807797 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:45:29.807877 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:29.808069 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 13:45:29.808162 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":71,"threads_started":1}
I20260924 13:45:29.808288 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:29.797908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39633 (local address 127.0.0.1:44960)
0924 13:45:29.798329 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:29.798365 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:29.798429 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:29.798751 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:29.798758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:29.799362 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:45:29.799452 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:29.799455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:29.800975 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:29.800976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:29.802076 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:29.802080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:29.802712 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:29.802713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:29.802795 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:29.803331 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:45:29.803336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:45:29.804994 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:45:29.806983 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:29.806990 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:29.806992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:29.807272 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:29.807499 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:29.807500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:29.807501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:29.807566 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:29.807867 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:45:29.807873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:45:29.807934 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 13:45:29.808018 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 13:45:29.808146 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":106,"threads_started":1}
I20260924 13:45:29.809377 3291 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b04239ee7ba4729bdf30464400dd7b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:45:09 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:44960
I20260924 13:45:29.810019 3383 tablet_replica.cc:330] stopping tablet replica
I20260924 13:45:29.810133 3383 raft_consensus.cc:2229] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:45:29.810232 3383 raft_consensus.cc:2258] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:45:29.810536 3383 ts_tablet_manager.cc:1905] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:45:29.813076 3383 ts_tablet_manager.cc:1918] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:45:29.813148 3383 log.cc:1196] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal/wals/4b04239ee7ba4729bdf30464400dd7b3
I20260924 13:45:29.813422 3383 ts_tablet_manager.cc:1939] T 4b04239ee7ba4729bdf30464400dd7b3 P 9b0553c5fa53410880e5e1232348dc2c: Deleting consensus metadata
I20260924 13:45:29.813957 3172 catalog_manager.cc:4841] TS 9b0553c5fa53410880e5e1232348dc2c (127.19.53.193:39633): tablet 4b04239ee7ba4729bdf30464400dd7b3 (table test-table [id=589f21d12a684ebb94751b67e79b17ac]) successfully deleted
W20260924 13:45:29.878391 3379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:45:29.878615 3379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:45:29.878639 3379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:45:29.879820 3379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:45:29.879859 3379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:45:29.879884 3379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:45:29.881368 3379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:36563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=36377
--tserver_master_addrs=127.19.53.254:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:45:29.881624 3379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:45:29.881860 3379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:45:29.887015 3388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:29.887038 3389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:29.887137 3391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:29.887310 3379 server_base.cc:1034] running on GCE node
I20260924 13:45:29.887535 3379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:45:29.887727 3379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:45:29.888897 3379 hybrid_clock.cc:648] HybridClock initialized: now 1790257529888886 us; error 30 us; skew 500 ppm
Sep 24 13:45:29 dist-test-slave-tfm9 krb5kdc[2600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257529, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:45:29.891430 3379 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:45:29.891856 3379 webserver.cc:460] Webserver started at http://127.19.53.194:36377/ using document root <none> and password file <none>
I20260924 13:45:29.892103 3379 fs_manager.cc:359] Metadata directory not provided
I20260924 13:45:29.892187 3379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:45:29.893558 3379 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:45:29.894174 3398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:29.894351 3379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:45:29.894457 3379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "f1aecb9292344cf4b8bb47a2671db0cb"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "513063f9ed0c52412e64a23dc93efb29"
server_key_iv: "760bf467403fdf7bc0fa56029f347b21"
server_key_version: "encryptionkey@0"
I20260924 13:45:29.894798 3379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:45:29.907476 3379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:45:29.910342 3379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:45:29.910495 3379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:45:29.910737 3379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:45:29.911096 3379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:45:29.911144 3379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:29.911180 3379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:45:29.911211 3379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:29.918591 3379 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:36563
I20260924 13:45:29.918675 3512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:36563 every 8 connection(s)
I20260924 13:45:29.918905 3379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:45:29 dist-test-slave-tfm9 krb5kdc[2600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257529, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:45:29.925945 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 3379
I20260924 13:45:29.926277 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:37809
--local_ip_for_outbound_sockets=127.19.53.195
--tserver_master_addrs=127.19.53.254:38623
--webserver_port=40867
--webserver_interface=127.19.53.195
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:45:29.929396 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:29.919646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:49401 (local address 127.19.53.254:38623)
0924 13:45:29.919695 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:29.919697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:29.919994 (+ 297us) server_negotiation.cc:407] Connection header received
0924 13:45:29.920485 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:29.920487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:29.920503 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:29.920562 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:45:29.921480 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:29.922077 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:29.923358 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:29.923569 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:29.923631 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:29.925861 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:45:29.925862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:45:29.925863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:45:29.925871 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:45:29.927763 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:29.928305 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:29.928306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:29.928306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:29.928356 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:29.928634 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:29.928634 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:29.928634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:29.928736 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:45:29.928786 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:29.929159 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 13:45:29.929255 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:45:29.929404 3515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:29.919651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:49401)
0924 13:45:29.919860 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:29.919897 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:29.920395 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:29.920621 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:29.920625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:29.921191 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:45:29.921380 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:29.921385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:29.922168 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:29.922169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:29.923257 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:29.923259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:29.923657 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:29.923658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:29.923760 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:29.924233 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:45:29.924240 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:45:29.925781 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:45:29.927871 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:29.927878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:29.927880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:29.928227 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:29.928424 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:29.928425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:29.928425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:29.928566 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:29.928821 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:45:29.928826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:45:29.928996 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 13:45:29.929124 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 13:45:29.929246 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":97,"threads_started":1}
I20260924 13:45:29.929884 3513 heartbeater.cc:347] Connected to a master server at 127.19.53.254:38623
I20260924 13:45:29.930011 3513 heartbeater.cc:464] Registering TS with master...
I20260924 13:45:29.930214 3513 heartbeater.cc:511] Master 127.19.53.254:38623 requested a full tablet report, sending...
I20260924 13:45:29.930667 3185 ts_manager.cc:194] Registered new tserver with Master: f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563)
I20260924 13:45:29.931059 3185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:49401
W20260924 13:45:30.009665 3516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:45:30.009869 3516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:45:30.009892 3516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:45:30.011068 3516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:45:30.011111 3516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:45:30.011137 3516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:45:30.012472 3516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:37809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=40867
--tserver_master_addrs=127.19.53.254:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:45:30.012666 3516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:45:30.012830 3516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:45:30.018060 3522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:30.018041 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:30.018041 3521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:30.018247 3516 server_base.cc:1034] running on GCE node
I20260924 13:45:30.018460 3516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:45:30.018700 3516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:45:30.019868 3516 hybrid_clock.cc:648] HybridClock initialized: now 1790257530019843 us; error 41 us; skew 500 ppm
Sep 24 13:45:30 dist-test-slave-tfm9 krb5kdc[2600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257530, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:45:30.022454 3516 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:45:30.022914 3516 webserver.cc:460] Webserver started at http://127.19.53.195:40867/ using document root <none> and password file <none>
I20260924 13:45:30.023144 3516 fs_manager.cc:359] Metadata directory not provided
I20260924 13:45:30.023236 3516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:45:30.024595 3516 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:45:30.025228 3531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:30.025463 3516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:45:30.025568 3516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "48e03d2048ab4d329f0b715d8e1a07e4"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "c481ed5f8a1c699999feff482ed1f8da"
server_key_iv: "b9dc0681dee9225d9a04a975211b814b"
server_key_version: "encryptionkey@0"
I20260924 13:45:30.025940 3516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:45:30.058205 3516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:45:30.060482 3516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:45:30.060671 3516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:45:30.060922 3516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:45:30.061244 3516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:45:30.061298 3516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:30.061338 3516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:45:30.061367 3516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:30.067422 3516 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:37809
I20260924 13:45:30.067479 3645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:37809 every 8 connection(s)
I20260924 13:45:30.067798 3516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:45:30.072331 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 3516
Sep 24 13:45:30 dist-test-slave-tfm9 krb5kdc[2600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257530, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:45:30.078416 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.068605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:38249 (local address 127.19.53.254:38623)
0924 13:45:30.068720 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:30.068723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:30.068921 (+ 198us) server_negotiation.cc:407] Connection header received
0924 13:45:30.069484 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:30.069485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:30.069496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:30.069549 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:45:30.070609 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.071094 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.072086 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.072281 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.072331 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.075091 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:45:30.075092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:45:30.075092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:45:30.075098 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:45:30.076692 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:30.077247 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:30.077248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:30.077248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:30.077343 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:30.077628 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:30.077629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:30.077629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:30.077766 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:45:30.077836 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:30.078215 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 13:45:30.078309 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 13:45:30.078621 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.068616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:38249)
0924 13:45:30.068794 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:30.068833 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:30.069393 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:30.069640 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:30.069643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:30.070331 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:45:30.070518 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:30.070522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.071171 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.071172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:30.071982 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:30.071984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.072363 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.072364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:30.072519 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.073155 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:45:30.073164 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:45:30.074987 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:45:30.076779 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:30.076787 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:30.076789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:30.077118 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:30.077421 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:30.077422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:30.077422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:30.077523 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:30.077857 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:45:30.077863 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:45:30.078061 (+ 198us) client_negotiation.cc:769] Sending connection context
0924 13:45:30.078188 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 13:45:30.078358 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260924 13:45:30.079284 3646 heartbeater.cc:347] Connected to a master server at 127.19.53.254:38623
I20260924 13:45:30.079398 3646 heartbeater.cc:464] Registering TS with master...
I20260924 13:45:30.079586 3646 heartbeater.cc:511] Master 127.19.53.254:38623 requested a full tablet report, sending...
I20260924 13:45:30.080022 3185 ts_manager.cc:194] Registered new tserver with Master: 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809)
I20260924 13:45:30.080119 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.073132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46882 (local address 127.19.53.254:38623)
0924 13:45:30.073254 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:30.073256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:30.073264 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:45:30.073284 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:30.073285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:30.073295 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:30.073353 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:45:30.074175 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.074835 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.075665 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.075878 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.075970 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.077251 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:45:30.077252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:45:30.077252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:45:30.077258 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:45:30.078991 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:30.079345 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:30.079346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:30.079347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:30.079406 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:30.079634 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:30.079635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:30.079635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:30.079758 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:45:30.079953 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:30.079990 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 13:45:30.080061 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260924 13:45:30.080466 3185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:38249
I20260924 13:45:30.081007 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:45:30.083777 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.081356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.19.53.254:38623)
0924 13:45:30.081395 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:30.081397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:30.081468 (+ 71us) server_negotiation.cc:407] Connection header received
0924 13:45:30.081558 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:30.081558 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:30.081591 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:30.081673 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:45:30.081770 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.082289 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.083033 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.083260 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.083337 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.083512 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:45:30.083563 (+ 51us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 13:45:30.083712 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46884: token expired
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 13:45:30.083827 3650 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46884: token expired
W20260924 13:45:30.083966 3115 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.19.53.254:38623)
I20260924 13:45:30.089661 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.084225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.19.53.254:38623)
0924 13:45:30.084280 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:30.084281 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:30.084332 (+ 51us) server_negotiation.cc:407] Connection header received
0924 13:45:30.084401 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:30.084402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:30.084410 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:30.084463 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:45:30.084912 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.085344 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.086126 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.086321 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.086402 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.087434 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:45:30.087435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:45:30.087436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:45:30.087442 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:45:30.088563 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:30.088907 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:30.088908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:30.088908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:30.088959 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:30.089175 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:30.089176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:30.089176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:30.089280 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:45:30.089388 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:30.089493 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 13:45:30.089570 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:45:30.090796 3185 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:45:30.092325 3185 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:45:30.094527 3291 tablet_service.cc:1467] Processing CreateTablet for tablet a7cffdb803f348dea06edbddfb0bd96a (DEFAULT_TABLE table=test-table [id=5a9f22471e224ab3a48207fa232836cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:45:30.094729 3291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7cffdb803f348dea06edbddfb0bd96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:45:30.096249 3265 tablet_bootstrap.cc:492] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c: Bootstrap starting.
I20260924 13:45:30.096561 3265 tablet_bootstrap.cc:654] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c: Neither blocks nor log segments found. Creating new log.
I20260924 13:45:30.097075 3265 tablet_bootstrap.cc:492] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c: No bootstrap required, opened a new log
I20260924 13:45:30.097126 3265 ts_tablet_manager.cc:1397] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 13:45:30.097445 3265 raft_consensus.cc:348] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:30.097548 3265 raft_consensus.cc:374] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:45:30.097594 3265 raft_consensus.cc:729] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0553c5fa53410880e5e1232348dc2c, State: Initialized, Role: FOLLOWER
I20260924 13:45:30.097635 3265 consensus_queue.cc:260] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:30.097729 3265 ts_tablet_manager.cc:1428] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 13:45:30.098191 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.094207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:36563 (local address 127.0.0.1:52228)
0924 13:45:30.094240 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:30.094251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:30.094288 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:30.094741 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:30.094742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:30.094744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:45:30.094789 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:30.094790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.095962 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.095963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:30.096962 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:30.096965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.097944 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.097945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:30.098013 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.098037 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 13:45:30.098100 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 13:45:30.098145 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":18}
I20260924 13:45:30.098237 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.094332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.19.53.194:36563)
0924 13:45:30.094519 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:30.094523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:30.094542 (+ 19us) server_negotiation.cc:407] Connection header received
0924 13:45:30.094582 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:30.094584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:30.094607 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:30.094731 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:45:30.094839 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.095858 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.097037 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.097836 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.097951 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.098017 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:30.098113 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 13:45:30.098155 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
I20260924 13:45:30.099110 3447 tablet_service.cc:1467] Processing CreateTablet for tablet a7cffdb803f348dea06edbddfb0bd96a (DEFAULT_TABLE table=test-table [id=5a9f22471e224ab3a48207fa232836cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:45:30.099347 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.094303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:37809 (local address 127.0.0.1:43932)
0924 13:45:30.094683 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:30.094693 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:30.094748 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:30.095043 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:30.095043 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:30.095045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:45:30.095107 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:30.095109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.095950 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.095951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:30.097997 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:30.097999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.099092 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.099093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:30.099155 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.099158 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:45:30.099248 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 13:45:30.099284 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":103,"threads_started":1}
I20260924 13:45:30.099380 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.094476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.19.53.195:37809)
0924 13:45:30.094602 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:30.094606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:30.094853 (+ 247us) server_negotiation.cc:407] Connection header received
0924 13:45:30.094892 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:30.094894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:30.094913 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:30.094964 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:45:30.095180 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.095852 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.098102 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.098910 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.099000 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.099053 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:30.099266 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 13:45:30.099304 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
I20260924 13:45:30.099489 3447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7cffdb803f348dea06edbddfb0bd96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:45:30.099959 3579 tablet_service.cc:1467] Processing CreateTablet for tablet a7cffdb803f348dea06edbddfb0bd96a (DEFAULT_TABLE table=test-table [id=5a9f22471e224ab3a48207fa232836cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:45:30.100242 3579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7cffdb803f348dea06edbddfb0bd96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:45:30.102543 3656 tablet_bootstrap.cc:492] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb: Bootstrap starting.
I20260924 13:45:30.103077 3656 tablet_bootstrap.cc:654] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb: Neither blocks nor log segments found. Creating new log.
I20260924 13:45:30.103348 3657 tablet_bootstrap.cc:492] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4: Bootstrap starting.
I20260924 13:45:30.103740 3656 log.cc:824] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb: Log is configured to *not* fsync() on all Append() calls
I20260924 13:45:30.104069 3657 tablet_bootstrap.cc:654] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4: Neither blocks nor log segments found. Creating new log.
I20260924 13:45:30.104666 3657 log.cc:824] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4: Log is configured to *not* fsync() on all Append() calls
I20260924 13:45:30.104698 3656 tablet_bootstrap.cc:492] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb: No bootstrap required, opened a new log
I20260924 13:45:30.104758 3656 ts_tablet_manager.cc:1397] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:45:30.105406 3657 tablet_bootstrap.cc:492] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4: No bootstrap required, opened a new log
I20260924 13:45:30.105479 3657 ts_tablet_manager.cc:1397] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:45:30.106745 3656 raft_consensus.cc:348] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:30.106926 3656 raft_consensus.cc:374] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:45:30.106963 3656 raft_consensus.cc:729] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1aecb9292344cf4b8bb47a2671db0cb, State: Initialized, Role: FOLLOWER
I20260924 13:45:30.107084 3656 consensus_queue.cc:260] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:30.107386 3656 ts_tablet_manager.cc:1428] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:45:30.107398 3513 heartbeater.cc:503] Master 127.19.53.254:38623 was elected leader, sending a full tablet report...
I20260924 13:45:30.107437 3657 raft_consensus.cc:348] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:30.107640 3657 raft_consensus.cc:374] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:45:30.107690 3657 raft_consensus.cc:729] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e03d2048ab4d329f0b715d8e1a07e4, State: Initialized, Role: FOLLOWER
I20260924 13:45:30.107803 3657 consensus_queue.cc:260] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:30.108079 3657 ts_tablet_manager.cc:1428] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:45:30.108106 3646 heartbeater.cc:503] Master 127.19.53.254:38623 was elected leader, sending a full tablet report...
I20260924 13:45:30.123185 3661 raft_consensus.cc:482] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:45:30.123303 3661 raft_consensus.cc:504] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:30.124012 3661 leader_election.cc:288] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563), 9b0553c5fa53410880e5e1232348dc2c (127.19.53.193:39633)
I20260924 13:45:30.127007 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.124190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:54729 (local address 127.19.53.193:39633)
0924 13:45:30.124251 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:30.124253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:30.124262 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:45:30.124289 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:30.124290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:30.124320 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:30.124393 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:45:30.124583 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.125102 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.126196 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.126660 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.126771 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.126853 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:30.126900 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 13:45:30.126942 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 13:45:30.127007 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.123963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39633 (local address 127.19.53.195:54729)
0924 13:45:30.124022 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:30.124036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:30.124099 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:30.124414 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:30.124415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:30.124417 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:45:30.124504 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:30.124506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.125197 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.125198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:30.126100 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:30.126102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.126753 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.126754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:30.126833 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.126836 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:45:30.126893 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 13:45:30.126933 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 13:45:30.127125 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.123970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:36563 (local address 127.19.53.195:45993)
0924 13:45:30.124216 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:30.124228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:30.124292 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:30.124455 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:30.124456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:30.124458 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:45:30.124562 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:30.124564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.125247 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.125248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:30.126131 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:30.126133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.126825 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.126826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:30.126905 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.126908 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:45:30.126978 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:45:30.127016 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":58,"thread_start_us":119,"threads_started":1}
I20260924 13:45:30.127112 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.124050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:45993 (local address 127.19.53.194:36563)
0924 13:45:30.124099 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:30.124101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:30.124307 (+ 206us) server_negotiation.cc:407] Connection header received
0924 13:45:30.124339 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:30.124340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:30.124351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:30.124429 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:45:30.124647 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.125097 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.126235 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.126730 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.126809 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.126817 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:30.126982 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 13:45:30.127042 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 13:45:30.127564 3467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7cffdb803f348dea06edbddfb0bd96a" candidate_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" is_pre_election: true
I20260924 13:45:30.127556 3324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7cffdb803f348dea06edbddfb0bd96a" candidate_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b0553c5fa53410880e5e1232348dc2c" is_pre_election: true
I20260924 13:45:30.127718 3324 raft_consensus.cc:2454] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e03d2048ab4d329f0b715d8e1a07e4 in term 0.
I20260924 13:45:30.127728 3467 raft_consensus.cc:2454] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e03d2048ab4d329f0b715d8e1a07e4 in term 0.
I20260924 13:45:30.127948 3532 leader_election.cc:302] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e03d2048ab4d329f0b715d8e1a07e4, f1aecb9292344cf4b8bb47a2671db0cb; no voters:
I20260924 13:45:30.128124 3661 raft_consensus.cc:2781] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:45:30.128212 3661 raft_consensus.cc:482] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:45:30.128242 3661 raft_consensus.cc:3037] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:45:30.128933 3661 raft_consensus.cc:504] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:30.129081 3661 leader_election.cc:288] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [CANDIDATE]: Term 1 election: Requested vote from peers f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563), 9b0553c5fa53410880e5e1232348dc2c (127.19.53.193:39633)
I20260924 13:45:30.129268 3467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7cffdb803f348dea06edbddfb0bd96a" candidate_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1aecb9292344cf4b8bb47a2671db0cb"
I20260924 13:45:30.129284 3324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7cffdb803f348dea06edbddfb0bd96a" candidate_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b0553c5fa53410880e5e1232348dc2c"
I20260924 13:45:30.129355 3324 raft_consensus.cc:3037] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:45:30.129354 3467 raft_consensus.cc:3037] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:45:30.130263 3324 raft_consensus.cc:2454] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e03d2048ab4d329f0b715d8e1a07e4 in term 1.
I20260924 13:45:30.130263 3467 raft_consensus.cc:2454] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e03d2048ab4d329f0b715d8e1a07e4 in term 1.
I20260924 13:45:30.130439 3535 leader_election.cc:302] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e03d2048ab4d329f0b715d8e1a07e4, 9b0553c5fa53410880e5e1232348dc2c; no voters:
I20260924 13:45:30.130519 3661 raft_consensus.cc:2781] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:45:30.130654 3661 raft_consensus.cc:686] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [term 1 LEADER]: Becoming Leader. State: Replica: 48e03d2048ab4d329f0b715d8e1a07e4, State: Running, Role: LEADER
I20260924 13:45:30.130757 3661 consensus_queue.cc:237] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:30.131675 3184 catalog_manager.cc:5495] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195). New cstate: current_term: 1 leader_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } health_report { overall_health: UNKNOWN } } }
I20260924 13:45:30.146130 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.143900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.19.53.195:37809)
0924 13:45:30.143965 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:30.143967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:30.143975 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:45:30.144012 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:30.144012 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:30.144032 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:30.144108 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:45:30.144220 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.144691 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.145374 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:30.145547 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:30.145610 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:30.145744 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:45:30.145840 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:45:30.145899 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:30.146055 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 13:45:30.146074 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:45:30.147913 3467 raft_consensus.cc:1261] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb [term 1 FOLLOWER]: Refusing update from remote peer 48e03d2048ab4d329f0b715d8e1a07e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:45:30.147915 3324 raft_consensus.cc:1261] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Refusing update from remote peer 48e03d2048ab4d329f0b715d8e1a07e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:45:30.148303 3661 consensus_queue.cc:1035] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:45:30.148465 3663 consensus_queue.cc:1035] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:45:30.150360 3666 mvcc.cc:205] Tried to move back new op lower bound from 7332894843483217920 to 7332894843416399872. Current Snapshot: MvccSnapshot[applied={T|T < 7332894843483217920}]
I20260924 13:45:30.152347 3668 mvcc.cc:205] Tried to move back new op lower bound from 7332894843483217920 to 7332894843416399872. Current Snapshot: MvccSnapshot[applied={T|T < 7332894843483217920 or (T in {7332894843483217920})}]
I20260924 13:45:30.152657 3184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:45:30.152750 3184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:45:30.153525 3184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Sending DeleteTablet for 3 replicas of tablet a7cffdb803f348dea06edbddfb0bd96a
I20260924 13:45:30.153823 3291 tablet_service.cc:1514] Processing DeleteTablet for tablet a7cffdb803f348dea06edbddfb0bd96a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:45:30 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:44960
I20260924 13:45:30.153864 3579 tablet_service.cc:1514] Processing DeleteTablet for tablet a7cffdb803f348dea06edbddfb0bd96a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:45:30 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:43932
I20260924 13:45:30.153937 3383 tablet_replica.cc:330] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c: stopping tablet replica
I20260924 13:45:30.153898 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 3223
I20260924 13:45:30.153986 3383 raft_consensus.cc:2229] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:45:30.154043 3383 raft_consensus.cc:2258] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:45:30.154573 3447 tablet_service.cc:1514] Processing DeleteTablet for tablet a7cffdb803f348dea06edbddfb0bd96a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:45:30 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:52228
I20260924 13:45:30.154796 3676 tablet_replica.cc:330] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb: stopping tablet replica
I20260924 13:45:30.154922 3676 raft_consensus.cc:2229] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:45:30.155069 3676 raft_consensus.cc:2258] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:45:30.155736 3676 ts_tablet_manager.cc:1905] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:45:30.157399 3676 ts_tablet_manager.cc:1918] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:45:30.157490 3676 log.cc:1196] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/a7cffdb803f348dea06edbddfb0bd96a
I20260924 13:45:30.157768 3676 ts_tablet_manager.cc:1939] T a7cffdb803f348dea06edbddfb0bd96a P f1aecb9292344cf4b8bb47a2671db0cb: Deleting consensus metadata
I20260924 13:45:30.158022 3675 tablet_replica.cc:330] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4: stopping tablet replica
I20260924 13:45:30.158169 3675 raft_consensus.cc:2229] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:45:30.158226 3169 catalog_manager.cc:4841] TS f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563): tablet a7cffdb803f348dea06edbddfb0bd96a (table test-table [id=5a9f22471e224ab3a48207fa232836cc]) successfully deleted
I20260924 13:45:30.158365 3675 raft_consensus.cc:2258] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:45:30.158741 3675 ts_tablet_manager.cc:1905] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:45:30.160326 3675 ts_tablet_manager.cc:1918] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:45:30.160411 3675 log.cc:1196] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/a7cffdb803f348dea06edbddfb0bd96a
W20260924 13:45:30.160662 3172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:39633 (error 108)
I20260924 13:45:30.160887 3675 ts_tablet_manager.cc:1939] T a7cffdb803f348dea06edbddfb0bd96a P 48e03d2048ab4d329f0b715d8e1a07e4: Deleting consensus metadata
I20260924 13:45:30.160928 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 3379
I20260924 13:45:30.162364 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:30.162232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:39633 (local address 127.0.0.1:44974)
0924 13:45:30.162284 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:30.162330 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:39633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 13:45:30.162593 3169 catalog_manager.cc:4841] TS 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809): tablet a7cffdb803f348dea06edbddfb0bd96a (table test-table [id=5a9f22471e224ab3a48207fa232836cc]) successfully deleted
W20260924 13:45:30.163400 3172 catalog_manager.cc:4584] TS 9b0553c5fa53410880e5e1232348dc2c (127.19.53.193:39633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7cffdb803f348dea06edbddfb0bd96a: Network error: Client connection negotiation failed: client connection to 127.19.53.193:39633: connect: Connection refused (error 111)
I20260924 13:45:30.167995 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 3516
I20260924 13:45:30.172971 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 3152
I20260924 13:45:50.178300 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:38623
--webserver_interface=127.19.53.254
--webserver_port=34109
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:45:50.258307 3678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:45:50.258502 3678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:45:50.258527 3678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:45:50.259588 3678 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 13:45:50.259617 3678 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 13:45:50.259629 3678 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 13:45:50.259640 3678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:45:50.259654 3678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:45:50.259666 3678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:45:50.259678 3678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:45:50.259694 3678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:45:50.261051 3678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:38623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:45:50.261234 3678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:45:50.261387 3678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:45:50.266325 3684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:50.266333 3686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:50.266366 3683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:50.266363 3678 server_base.cc:1034] running on GCE node
I20260924 13:45:50.266784 3678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:45:50.267081 3678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:45:50.268256 3678 hybrid_clock.cc:648] HybridClock initialized: now 1790257550268245 us; error 25 us; skew 500 ppm
Sep 24 13:45:50 dist-test-slave-tfm9 krb5kdc[2600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257550, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:45:50.270593 3678 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:45:50.270951 3678 webserver.cc:460] Webserver started at http://127.19.53.254:34109/ using document root <none> and password file <none>
I20260924 13:45:50.271140 3678 fs_manager.cc:359] Metadata directory not provided
I20260924 13:45:50.271181 3678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:45:50.272377 3678 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:45:50.272989 3693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:50.273154 3678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:45:50.273231 3678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "e94886c9b8574815a47a3613ed7a4918"
format_stamp: "Formatted at 2026-09-24 13:45:08 on dist-test-slave-tfm9"
server_key: "ec6615a99739cd92f70d4608aae250a9"
server_key_iv: "c83d6869399f8c2899b96439c29b6c00"
server_key_version: "encryptionkey@0"
I20260924 13:45:50.273581 3678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:45:50.295164 3678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:45:50.297494 3678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:45:50.297601 3678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:45:50.301357 3678 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:38623
I20260924 13:45:50.301409 3745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:38623 every 8 connection(s)
I20260924 13:45:50.301803 3678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:45:50.303121 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 3678
I20260924 13:45:50.303419 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:39633
--local_ip_for_outbound_sockets=127.19.53.193
--tserver_master_addrs=127.19.53.254:38623
--webserver_port=39601
--webserver_interface=127.19.53.193
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:45:50.303416 3746 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:45:50.304093 3746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Bootstrap starting.
I20260924 13:45:50.312402 3746 log.cc:824] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Log is configured to *not* fsync() on all Append() calls
I20260924 13:45:50.314597 3746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:45:50.314888 3746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: Bootstrap complete.
I20260924 13:45:50.317269 3746 raft_consensus.cc:348] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } }
I20260924 13:45:50.317509 3746 raft_consensus.cc:729] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94886c9b8574815a47a3613ed7a4918, State: Initialized, Role: FOLLOWER
I20260924 13:45:50.317728 3746 consensus_queue.cc:260] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } }
I20260924 13:45:50.317833 3746 raft_consensus.cc:388] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:45:50.317884 3746 raft_consensus.cc:482] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:45:50.317955 3746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 2 FOLLOWER]: Advancing to term 3
I20260924 13:45:50.318997 3746 raft_consensus.cc:504] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } }
I20260924 13:45:50.319113 3746 leader_election.cc:302] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e94886c9b8574815a47a3613ed7a4918; no voters:
I20260924 13:45:50.319329 3746 leader_election.cc:288] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 13:45:50.319363 3750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 13:45:50.319487 3750 raft_consensus.cc:686] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [term 3 LEADER]: Becoming Leader. State: Replica: e94886c9b8574815a47a3613ed7a4918, State: Running, Role: LEADER
I20260924 13:45:50.319592 3746 sys_catalog.cc:564] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:45:50.319633 3750 consensus_queue.cc:237] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } }
I20260924 13:45:50.320024 3751 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e94886c9b8574815a47a3613ed7a4918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } } }
I20260924 13:45:50.320127 3751 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: This master's current role is: LEADER
I20260924 13:45:50.320024 3750 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e94886c9b8574815a47a3613ed7a4918. Latest consensus state: current_term: 3 leader_uuid: "e94886c9b8574815a47a3613ed7a4918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94886c9b8574815a47a3613ed7a4918" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38623 } } }
I20260924 13:45:50.320329 3750 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918 [sys.catalog]: This master's current role is: LEADER
I20260924 13:45:50.321314 3766 catalog_manager.cc:1259] Loaded cluster ID: b10eea66bf844ac588ec3917df4bb91b
I20260924 13:45:50.321362 3766 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: loading cluster ID for follower catalog manager: success
I20260924 13:45:50.322528 3766 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e94886c9b8574815a47a3613ed7a4918: acquiring CA information for follower catalog manager: success
F20260924 13:45:50.322906 3766 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efe9bce39ad google::LogMessage::Fail() at ??:0
@ 0x7efe9bce688a google::LogMessage::SendToLog() at ??:0
@ 0x7efe9bce34de google::LogMessage::Flush() at ??:0
@ 0x7efe9bce4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efe9e450104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7efea3c3c6b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7efea3c3c989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7efea3c3ce4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7efea3c3f8c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7efe9dba6954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7efe9ea0a6db start_thread at ??:0
@ 0x7efe9a7e161f clone at ??:0
W20260924 13:45:50.400569 3748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:45:50.400756 3748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:45:50.400779 3748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:45:50.401953 3748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:45:50.402004 3748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:45:50.402029 3748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:45:50.403486 3748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:39633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=39601
--tserver_master_addrs=127.19.53.254:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:45:50.403707 3748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:45:50.403959 3748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:45:50.408865 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:50.408885 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:50.408872 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:50.409535 3748 server_base.cc:1034] running on GCE node
I20260924 13:45:50.409745 3748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:45:50.410044 3748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:45:50.411204 3748 hybrid_clock.cc:648] HybridClock initialized: now 1790257550411193 us; error 32 us; skew 500 ppm
Sep 24 13:45:50 dist-test-slave-tfm9 krb5kdc[2600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257550, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:45:50.413797 3748 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:45:50.414310 3748 webserver.cc:460] Webserver started at http://127.19.53.193:39601/ using document root <none> and password file <none>
I20260924 13:45:50.414566 3748 fs_manager.cc:359] Metadata directory not provided
I20260924 13:45:50.414636 3748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:45:50.416095 3748 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:45:50.416843 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:50.417086 3748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:45:50.417181 3748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "9b0553c5fa53410880e5e1232348dc2c"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "b917a2dc734fa7924c8a1b81ab8b2e3a"
server_key_iv: "7b004f3beb074ecaf6901306b33bcff8"
server_key_version: "encryptionkey@0"
I20260924 13:45:50.417642 3748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:45:50.452927 3748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:45:50.455466 3748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:45:50.455617 3748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:45:50.455871 3748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:45:50.456362 3801 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 13:45:50.457939 3748 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 13:45:50.457993 3748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 13:45:50.458025 3748 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 13:45:50.458976 3748 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 13:45:50.459022 3748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 13:45:50.459102 3801 tablet_bootstrap.cc:492] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c: Bootstrap starting.
I20260924 13:45:50.466243 3748 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:39633
I20260924 13:45:50.466310 3908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:39633 every 8 connection(s)
I20260924 13:45:50.466671 3748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:45:50.468972 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 3748
I20260924 13:45:50.469259 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:36563
--local_ip_for_outbound_sockets=127.19.53.194
--tserver_master_addrs=127.19.53.254:38623
--webserver_port=36377
--webserver_interface=127.19.53.194
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:45:50.470188 3801 log.cc:824] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c: Log is configured to *not* fsync() on all Append() calls
I20260924 13:45:50.472590 3801 tablet_bootstrap.cc:492] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:45:50.472875 3801 tablet_bootstrap.cc:492] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c: Bootstrap complete.
I20260924 13:45:50.473852 3801 ts_tablet_manager.cc:1397] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20260924 13:45:50.474943 3801 raft_consensus.cc:348] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:50.475200 3801 raft_consensus.cc:729] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0553c5fa53410880e5e1232348dc2c, State: Initialized, Role: FOLLOWER
I20260924 13:45:50.475348 3801 consensus_queue.cc:260] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:50.475629 3801 ts_tablet_manager.cc:1428] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20260924 13:45:50.536670 3911 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:45:50.467458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:59047)
0924 13:45:50.467638 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:50.467675 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:50.468102 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:50.536448 (+ 68346us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
W20260924 13:45:50.536872 3909 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:38623 (0 consecutive failures): Network error: Failed to ping master at 127.19.53.254:38623: Client connection negotiation failed: client connection to 127.19.53.254:38623: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 13:45:50.537249 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:50.537115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:44933)
0924 13:45:50.537176 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:50.537219 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 13:45:50.537636 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:50.537547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:57647)
0924 13:45:50.537597 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:50.537620 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 13:45:50.537762 3909 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:45:50.571123 3912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:45:50.571322 3912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:45:50.571347 3912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:45:50.572486 3912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:45:50.572528 3912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:45:50.572582 3912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:45:50.574069 3912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:36563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=36377
--tserver_master_addrs=127.19.53.254:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:45:50.574283 3912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:45:50.574519 3912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:45:50.579430 3920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:50.579453 3922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:50.579447 3919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:50.579703 3912 server_base.cc:1034] running on GCE node
I20260924 13:45:50.579890 3912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:45:50.580090 3912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:45:50.581251 3912 hybrid_clock.cc:648] HybridClock initialized: now 1790257550581233 us; error 32 us; skew 500 ppm
Sep 24 13:45:50 dist-test-slave-tfm9 krb5kdc[2600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257550, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:45:50.583659 3912 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:45:50.584137 3912 webserver.cc:460] Webserver started at http://127.19.53.194:36377/ using document root <none> and password file <none>
I20260924 13:45:50.584357 3912 fs_manager.cc:359] Metadata directory not provided
I20260924 13:45:50.584425 3912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:45:50.585846 3912 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:45:50.586467 3929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:50.586630 3912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:50.586732 3912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "f1aecb9292344cf4b8bb47a2671db0cb"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "513063f9ed0c52412e64a23dc93efb29"
server_key_iv: "760bf467403fdf7bc0fa56029f347b21"
server_key_version: "encryptionkey@0"
I20260924 13:45:50.587066 3912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:45:50.612812 3912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:45:50.614683 3912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:45:50.614828 3912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:45:50.615091 3912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:45:50.615408 3912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:45:50.615466 3912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:50.615507 3912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:45:50.615531 3912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:50.621531 3912 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:36563
I20260924 13:45:50.621578 4042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:36563 every 8 connection(s)
I20260924 13:45:50.621987 3912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:45:50.623083 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:50.622658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:46617)
0924 13:45:50.622829 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:50.622910 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
W20260924 13:45:50.623509 4043 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:38623 (0 consecutive failures): Network error: Failed to ping master at 127.19.53.254:38623: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
I20260924 13:45:50.623795 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:50.623709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:39495)
0924 13:45:50.623752 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:50.623778 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 13:45:50.624084 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:50.624008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:39089)
0924 13:45:50.624043 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:50.624064 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 13:45:50.624200 4043 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 13:45:50.625281 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 3912
I20260924 13:45:50.625517 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:37809
--local_ip_for_outbound_sockets=127.19.53.195
--tserver_master_addrs=127.19.53.254:38623
--webserver_port=40867
--webserver_interface=127.19.53.195
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:45:50.703433 4046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:45:50.703663 4046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:45:50.703699 4046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:45:50.704825 4046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:45:50.704895 4046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:45:50.704936 4046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:45:50.706362 4046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:37809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=40867
--tserver_master_addrs=127.19.53.254:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:45:50.706624 4046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:45:50.706845 4046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:45:50.711737 4051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:50.711737 4054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:45:50.711777 4052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:45:50.712044 4046 server_base.cc:1034] running on GCE node
I20260924 13:45:50.712296 4046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:45:50.712549 4046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:45:50.713711 4046 hybrid_clock.cc:648] HybridClock initialized: now 1790257550713699 us; error 42 us; skew 500 ppm
Sep 24 13:45:50 dist-test-slave-tfm9 krb5kdc[2600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257550, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:45:50.716274 4046 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:45:50.716719 4046 webserver.cc:460] Webserver started at http://127.19.53.195:40867/ using document root <none> and password file <none>
I20260924 13:45:50.717000 4046 fs_manager.cc:359] Metadata directory not provided
I20260924 13:45:50.717049 4046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:45:50.718557 4046 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:45:50.719167 4061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:50.719391 4046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:45:50.719517 4046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "48e03d2048ab4d329f0b715d8e1a07e4"
format_stamp: "Formatted at 2026-09-24 13:45:09 on dist-test-slave-tfm9"
server_key: "c481ed5f8a1c699999feff482ed1f8da"
server_key_iv: "b9dc0681dee9225d9a04a975211b814b"
server_key_version: "encryptionkey@0"
I20260924 13:45:50.719842 4046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:45:50.734436 4046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:45:50.736157 4046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:45:50.736264 4046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:45:50.736431 4046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:45:50.736724 4046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:45:50.736756 4046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:50.736778 4046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:45:50.736824 4046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:45:50.742643 4046 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:37809
I20260924 13:45:50.742745 4174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:37809 every 8 connection(s)
I20260924 13:45:50.743060 4046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:45:50.744365 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:50.743774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:57075)
0924 13:45:50.743967 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:50.744033 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
W20260924 13:45:50.744634 4175 heartbeater.cc:644] Failed to heartbeat to 127.19.53.254:38623 (0 consecutive failures): Network error: Failed to ping master at 127.19.53.254:38623: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
I20260924 13:45:50.744922 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:50.744810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:51339)
0924 13:45:50.744866 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:50.744899 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 13:45:50.745244 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:50.745154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:44019)
0924 13:45:50.745199 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:50.745226 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 13:45:50.745388 4175 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 13:45:50.750162 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 4046
I20260924 13:45:51.538389 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:51.538154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:52345)
0924 13:45:51.538285 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:51.538326 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260924 13:45:51.624699 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:51.624507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:55403)
0924 13:45:51.624617 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:51.624651 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260924 13:45:51.745898 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:51.745689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:53751)
0924 13:45:51.745797 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:51.745846 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":52,"threads_started":1}
I20260924 13:45:51.762105 4182 raft_consensus.cc:482] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:45:51.762255 4182 raft_consensus.cc:504] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:51.762604 4182 leader_election.cc:288] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563), 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809)
Sep 24 13:45:51 dist-test-slave-tfm9 krb5kdc[2600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257550, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.194@KRBTEST.COM
Sep 24 13:45:51 dist-test-slave-tfm9 krb5kdc[2600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257550, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.195@KRBTEST.COM
I20260924 13:45:51.773377 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:51.763062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:37809 (local address 127.19.53.193:53149)
0924 13:45:51.763169 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:51.763180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:51.763238 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:51.764024 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:51.764026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:51.764350 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:45:51.764535 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:51.764541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:51.765775 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:51.765776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:51.766604 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:51.766606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:51.767225 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:51.767226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:51.767340 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:51.767981 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:45:51.767986 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:45:51.769742 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:45:51.771878 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:51.771884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:51.771886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:51.772103 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:51.772387 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:51.772388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:51.772388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:51.772496 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:51.772874 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:45:51.772880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:45:51.773011 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 13:45:51.773118 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 13:45:51.773274 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":6,"thread_start_us":72,"threads_started":1}
I20260924 13:45:51.773403 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:51.763137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:53149 (local address 127.19.53.195:37809)
0924 13:45:51.763309 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:51.763319 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:51.763349 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:45:51.763781 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:51.763783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:51.763824 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:51.763982 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:45:51.764656 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:51.765682 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:51.766705 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:51.767118 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:51.767249 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:51.769846 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:45:51.769850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:45:51.769850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:45:51.769867 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:45:51.771748 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:51.772208 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:51.772211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:51.772212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:51.772298 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:51.772566 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:51.772567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:51.772567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:51.772794 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:45:51.772889 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:51.773170 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 13:45:51.773297 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
I20260924 13:45:51.773739 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:51.763003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:60981 (local address 127.19.53.194:36563)
0924 13:45:51.763278 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 13:45:51.763292 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 13:45:51.763337 (+ 45us) server_negotiation.cc:407] Connection header received
0924 13:45:51.763814 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:45:51.763816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:45:51.763866 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:45:51.764025 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:45:51.764788 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:51.765819 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:51.766705 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:45:51.767123 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:45:51.767252 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:51.769751 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:45:51.769760 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:45:51.769760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:45:51.769772 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:45:51.772050 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:51.772496 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:51.772500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:51.772500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:51.772604 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:45:51.772864 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:45:51.772865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:45:51.772865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:45:51.773112 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:45:51.773218 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 13:45:51.773446 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 13:45:51.773615 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":190,"threads_started":1}
I20260924 13:45:51.773622 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:51.762889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:36563 (local address 127.19.53.193:60981)
0924 13:45:51.762953 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:51.762967 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:45:51.763039 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:45:51.764029 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:45:51.764030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:45:51.764423 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:45:51.764536 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:51.764541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:51.765929 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:51.765930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:51.766602 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:45:51.766604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:45:51.767229 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:45:51.767230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:45:51.767335 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:45:51.767930 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:45:51.767938 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:45:51.769606 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:45:51.772176 (+ 2570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:51.772177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:51.772178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:51.772385 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:51.772690 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:45:51.772691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:45:51.772691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:45:51.772766 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:45:51.773197 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:45:51.773198 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:45:51.773252 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 13:45:51.773376 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 13:45:51.773488 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
W20260924 13:45:51.774794 3795 leader_election.cc:341] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809): Not found: Tablet not found: a7cffdb803f348dea06edbddfb0bd96a
W20260924 13:45:51.774864 3795 leader_election.cc:341] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563): Not found: Tablet not found: a7cffdb803f348dea06edbddfb0bd96a
I20260924 13:45:51.774883 3795 leader_election.cc:302] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0553c5fa53410880e5e1232348dc2c; no voters: 48e03d2048ab4d329f0b715d8e1a07e4, f1aecb9292344cf4b8bb47a2671db0cb
I20260924 13:45:51.774972 4182 raft_consensus.cc:2726] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:45:52.539064 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:52.538837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:56057)
0924 13:45:52.538965 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:52.539004 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260924 13:45:52.625283 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:52.625100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:48403)
0924 13:45:52.625211 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:52.625244 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260924 13:45:52.746613 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:52.746390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:42457)
0924 13:45:52.746538 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:52.746573 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260924 13:45:53.539877 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:53.539570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:58913)
0924 13:45:53.539793 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:53.539831 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":57,"threads_started":1}
I20260924 13:45:53.625995 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:53.625756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:58439)
0924 13:45:53.625882 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:53.625951 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260924 13:45:53.747246 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:53.747038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:36893)
0924 13:45:53.747177 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:53.747209 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260924 13:45:53.935555 4192 raft_consensus.cc:482] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:45:53.935647 4192 raft_consensus.cc:504] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:53.935802 4192 leader_election.cc:288] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563), 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809)
W20260924 13:45:53.936292 3795 leader_election.cc:341] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563): Not found: Tablet not found: a7cffdb803f348dea06edbddfb0bd96a
W20260924 13:45:53.936359 3795 leader_election.cc:341] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809): Not found: Tablet not found: a7cffdb803f348dea06edbddfb0bd96a
I20260924 13:45:53.936375 3795 leader_election.cc:302] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0553c5fa53410880e5e1232348dc2c; no voters: 48e03d2048ab4d329f0b715d8e1a07e4, f1aecb9292344cf4b8bb47a2671db0cb
I20260924 13:45:53.936465 4192 raft_consensus.cc:2726] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:45:54.540652 4193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:54.540423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:47377)
0924 13:45:54.540568 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:54.540605 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260924 13:45:54.626585 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:54.626389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:48623)
0924 13:45:54.626510 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:54.626544 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260924 13:45:54.747849 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:54.747655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:44433)
0924 13:45:54.747776 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:54.747809 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260924 13:45:55.541347 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:55.541133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:44259)
0924 13:45:55.541262 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:55.541300 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260924 13:45:55.561757 4197 raft_consensus.cc:482] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:45:55.561852 4197 raft_consensus.cc:504] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:55.562083 4197 leader_election.cc:288] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563), 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809)
W20260924 13:45:55.562518 3795 leader_election.cc:341] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563): Not found: Tablet not found: a7cffdb803f348dea06edbddfb0bd96a
W20260924 13:45:55.562628 3795 leader_election.cc:341] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809): Not found: Tablet not found: a7cffdb803f348dea06edbddfb0bd96a
I20260924 13:45:55.562676 3795 leader_election.cc:302] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0553c5fa53410880e5e1232348dc2c; no voters: 48e03d2048ab4d329f0b715d8e1a07e4, f1aecb9292344cf4b8bb47a2671db0cb
I20260924 13:45:55.562796 4197 raft_consensus.cc:2726] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:45:55.627264 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:55.627049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:47017)
0924 13:45:55.627182 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:55.627221 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260924 13:45:55.748523 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:55.748283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:48713)
0924 13:45:55.748444 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:55.748479 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260924 13:45:56.542162 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:56.541901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:58587)
0924 13:45:56.542065 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:56.542113 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":61,"threads_started":1}
I20260924 13:45:56.628057 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:56.627840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:55121)
0924 13:45:56.627966 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:56.628012 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260924 13:45:56.749251 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:56.749052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:55177)
0924 13:45:56.749170 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:56.749211 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260924 13:45:57.134089 4203 raft_consensus.cc:482] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:45:57.134183 4203 raft_consensus.cc:504] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:45:57.134310 4203 leader_election.cc:288] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563), 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809)
W20260924 13:45:57.134770 3795 leader_election.cc:341] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809): Not found: Tablet not found: a7cffdb803f348dea06edbddfb0bd96a
W20260924 13:45:57.134837 3795 leader_election.cc:341] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563): Not found: Tablet not found: a7cffdb803f348dea06edbddfb0bd96a
I20260924 13:45:57.134855 3795 leader_election.cc:302] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0553c5fa53410880e5e1232348dc2c; no voters: 48e03d2048ab4d329f0b715d8e1a07e4, f1aecb9292344cf4b8bb47a2671db0cb
I20260924 13:45:57.134977 4203 raft_consensus.cc:2726] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:45:57.542778 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:57.542587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:39865)
0924 13:45:57.542698 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:57.542731 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260924 13:45:57.628747 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:57.628525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:33641)
0924 13:45:57.628662 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:57.628694 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260924 13:45:57.750025 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:57.749741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:38627)
0924 13:45:57.749884 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:57.749963 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260924 13:45:58.543411 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:58.543212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:54699)
0924 13:45:58.543331 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:58.543365 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260924 13:45:58.629364 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:58.629174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:50889)
0924 13:45:58.629285 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:58.629319 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260924 13:45:58.750608 4209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:58.750421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:38091)
0924 13:45:58.750535 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:58.750569 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260924 13:45:59.544102 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:59.543892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:58549)
0924 13:45:59.544015 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:59.544056 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260924 13:45:59.630010 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:59.629761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:37177)
0924 13:45:59.629882 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:59.629967 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260924 13:45:59.751240 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:45:59.751043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:43981)
0924 13:45:59.751161 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 13:45:59.751197 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260924 13:46:00.544839 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:00.544614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:33597)
0924 13:46:00.544751 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:00.544792 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260924 13:46:00.630622 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:00.630428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:40363)
0924 13:46:00.630543 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:00.630579 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260924 13:46:00.751894 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:00.751692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:55145)
0924 13:46:00.751808 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:00.751843 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260924 13:46:01.545593 4216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:01.545361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:51297)
0924 13:46:01.545499 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:01.545547 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260924 13:46:01.631289 4217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:01.631087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:32975)
0924 13:46:01.631206 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:01.631241 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260924 13:46:01.752579 4218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:01.752348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:43647)
0924 13:46:01.752494 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:01.752532 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260924 13:46:02.546408 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:02.546125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:51697)
0924 13:46:02.546276 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:02.546317 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260924 13:46:02.631932 4220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:02.631727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:43657)
0924 13:46:02.631845 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:02.631878 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260924 13:46:02.753291 4221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:02.753071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:34213)
0924 13:46:02.753207 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:02.753245 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
I20260924 13:46:03.010977 4222 raft_consensus.cc:482] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:46:03.011096 4222 raft_consensus.cc:504] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1aecb9292344cf4b8bb47a2671db0cb" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36563 } } peers { permanent_uuid: "48e03d2048ab4d329f0b715d8e1a07e4" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 37809 } } peers { permanent_uuid: "9b0553c5fa53410880e5e1232348dc2c" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 39633 } }
I20260924 13:46:03.011291 4222 leader_election.cc:288] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563), 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809)
W20260924 13:46:03.011662 3795 leader_election.cc:341] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48e03d2048ab4d329f0b715d8e1a07e4 (127.19.53.195:37809): Not found: Tablet not found: a7cffdb803f348dea06edbddfb0bd96a
W20260924 13:46:03.011732 3795 leader_election.cc:341] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1aecb9292344cf4b8bb47a2671db0cb (127.19.53.194:36563): Not found: Tablet not found: a7cffdb803f348dea06edbddfb0bd96a
I20260924 13:46:03.011749 3795 leader_election.cc:302] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0553c5fa53410880e5e1232348dc2c; no voters: 48e03d2048ab4d329f0b715d8e1a07e4, f1aecb9292344cf4b8bb47a2671db0cb
I20260924 13:46:03.011819 4222 raft_consensus.cc:2726] T a7cffdb803f348dea06edbddfb0bd96a P 9b0553c5fa53410880e5e1232348dc2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:46:03.547173 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:03.546917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:45293)
0924 13:46:03.547062 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:03.547105 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260924 13:46:03.632651 4224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:03.632438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:53571)
0924 13:46:03.632569 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:03.632603 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260924 13:46:03.754031 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:03.753797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:33257)
0924 13:46:03.753943 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:03.753984 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260924 13:46:04.548007 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:04.547708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:55467)
0924 13:46:04.547892 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:04.547943 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260924 13:46:04.633339 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:04.633077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:39827)
0924 13:46:04.633203 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:04.633240 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260924 13:46:04.754727 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:04.754481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:44127)
0924 13:46:04.754639 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:04.754680 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260924 13:46:05.548877 4229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:05.548607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.193:38599)
0924 13:46:05.548784 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:05.548829 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":62,"threads_started":1}
I20260924 13:46:05.634059 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:05.633807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.194:58419)
0924 13:46:05.633983 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:05.634016 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260924 13:46:05.755523 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:05.755272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38623 (local address 127.19.53.195:38383)
0924 13:46:05.755442 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:05.755478 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.254:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
I20260924 13:46:05.760727 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 3748
I20260924 13:46:05.765630 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 3912
I20260924 13:46:05.770327 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 4046
2026-09-24T13:46:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58162 ms)
[----------] 30 tests from SecurityITest (230358 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:05 dist-test-slave-tfm9 krb5kdc[4234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:05 dist-test-slave-tfm9 krb5kdc[4234](info): set up 2 sockets
Sep 24 13:46:05 dist-test-slave-tfm9 krb5kdc[4234](info): commencing operation
krb5kdc: starting...
W20260924 13:46:06.819196 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:06 dist-test-slave-tfm9 krb5kdc[4234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:46:06.932822 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:35063
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:35063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:07.013195 4247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:07.013438 4247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:07.013473 4247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:07.014648 4247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:07.014714 4247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:07.014739 4247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:07.014758 4247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:07.014788 4247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:07.016222 4247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:35063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:35063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:07.016502 4247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:07.016718 4247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:46:07.021847 4247 server_base.cc:1034] running on GCE node
W20260924 13:46:07.021804 4255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:07.021790 4252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:07.021821 4253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:07.022406 4247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:07.022699 4247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:07.023871 4247 hybrid_clock.cc:648] HybridClock initialized: now 1790257567023865 us; error 36 us; skew 500 ppm
Sep 24 13:46:07 dist-test-slave-tfm9 krb5kdc[4234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257567, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:07.026391 4247 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:46:07.026822 4247 webserver.cc:460] Webserver started at http://127.19.53.254:41169/ using document root <none> and password file <none>
I20260924 13:46:07.027027 4247 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:07.027069 4247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:07.027153 4247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:07.028115 4247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "9bd90dd848a2494d8f4d31d21af94d67"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "6a7c92c2a5dcb73841c8f7d3ecb830a9"
server_key_iv: "d837ecc2811388b434a8b534c5a186d0"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.028414 4247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "9bd90dd848a2494d8f4d31d21af94d67"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "6a7c92c2a5dcb73841c8f7d3ecb830a9"
server_key_iv: "d837ecc2811388b434a8b534c5a186d0"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.029595 4247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 13:46:07.030319 4262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:07.030501 4247 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:07.030586 4247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "9bd90dd848a2494d8f4d31d21af94d67"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "6a7c92c2a5dcb73841c8f7d3ecb830a9"
server_key_iv: "d837ecc2811388b434a8b534c5a186d0"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.030642 4247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:07.042929 4247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:07.045162 4247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:07.045270 4247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:07.048955 4247 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:35063
I20260924 13:46:07.049015 4314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:35063 every 8 connection(s)
I20260924 13:46:07.049333 4247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:07.050330 4315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:07.052706 4315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67: Bootstrap starting.
I20260924 13:46:07.053308 4315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:07.053604 4315 log.cc:824] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:07.054373 4315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67: No bootstrap required, opened a new log
I20260924 13:46:07.056043 4315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd90dd848a2494d8f4d31d21af94d67" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35063 } }
I20260924 13:46:07.056254 4315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:07.056306 4315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd90dd848a2494d8f4d31d21af94d67, State: Initialized, Role: FOLLOWER
I20260924 13:46:07.056468 4315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd90dd848a2494d8f4d31d21af94d67" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35063 } }
I20260924 13:46:07.056574 4315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:07.056623 4315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:07.056680 4315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:07.057189 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 4247
I20260924 13:46:07.057276 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:07.057300 4315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd90dd848a2494d8f4d31d21af94d67" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35063 } }
I20260924 13:46:07.057415 4315 leader_election.cc:302] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bd90dd848a2494d8f4d31d21af94d67; no voters:
I20260924 13:46:07.057492 19671 external_mini_cluster.cc:1428] Setting key 4056b8e88ff69d126be2ddf9c6921a83
I20260924 13:46:07.057619 4315 leader_election.cc:288] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:07.057650 4318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:07.057847 4318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [term 1 LEADER]: Becoming Leader. State: Replica: 9bd90dd848a2494d8f4d31d21af94d67, State: Running, Role: LEADER
I20260924 13:46:07.057998 4315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:07.057981 4318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd90dd848a2494d8f4d31d21af94d67" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35063 } }
I20260924 13:46:07.058362 4318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bd90dd848a2494d8f4d31d21af94d67. Latest consensus state: current_term: 1 leader_uuid: "9bd90dd848a2494d8f4d31d21af94d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd90dd848a2494d8f4d31d21af94d67" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35063 } } }
I20260924 13:46:07.058526 4318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:07.058514 4319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bd90dd848a2494d8f4d31d21af94d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd90dd848a2494d8f4d31d21af94d67" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35063 } } }
I20260924 13:46:07.058763 4319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:07.059154 4324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:07.059738 4324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:07.061323 4324 catalog_manager.cc:1347] Generated new cluster ID: 87205a24201e4047ab5cebd0c292e55d
I20260924 13:46:07.061378 4324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:46:07 dist-test-slave-tfm9 krb5kdc[4234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:07.070283 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.058205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58224 (local address 127.19.53.254:35063)
0924 13:46:07.058409 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.058422 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.058907 (+ 485us) server_negotiation.cc:407] Connection header received
0924 13:46:07.059028 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.059036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.059099 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.059245 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:07.060144 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.061510 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.062633 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.063305 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.063426 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.066414 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:07.066418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:07.066419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:07.066431 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:07.068518 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:07.068961 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:07.068964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:07.068964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:07.069092 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:07.069320 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:07.069321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:07.069321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:07.069619 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:07.069774 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.069880 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.070037 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260924 13:46:07.074337 4324 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:07.075022 4324 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:07.079788 4324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67: Generated new TSK 0
I20260924 13:46:07.079964 4324 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:46:07.112164 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:35063
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:07.192230 4341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:07.192461 4341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:07.192497 4341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:07.193634 4341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:07.193697 4341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:07.193742 4341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:46:07.195188 4341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:35063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:07.195466 4341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:07.195691 4341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:07.200673 4346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:07.200685 4349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:07.200898 4347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:07.200899 4341 server_base.cc:1034] running on GCE node
I20260924 13:46:07.201149 4341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:07.201414 4341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:07.202564 4341 hybrid_clock.cc:648] HybridClock initialized: now 1790257567202550 us; error 30 us; skew 500 ppm
Sep 24 13:46:07 dist-test-slave-tfm9 krb5kdc[4234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257567, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:07.204962 4341 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:46:07.205425 4341 webserver.cc:460] Webserver started at http://127.19.53.193:33895/ using document root <none> and password file <none>
I20260924 13:46:07.205643 4341 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:07.205708 4341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:07.205868 4341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:07.206964 4341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "4679b5d6c14142838c3f511134c53cbf"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "28c4f2b33728864f080a488db8314e34"
server_key_iv: "4f250f8c528fc8c405891ed4bcf08ab6"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.207346 4341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "4679b5d6c14142838c3f511134c53cbf"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "28c4f2b33728864f080a488db8314e34"
server_key_iv: "4f250f8c528fc8c405891ed4bcf08ab6"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.208602 4341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:46:07.209295 4356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:07.209462 4341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:07.209558 4341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "4679b5d6c14142838c3f511134c53cbf"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "28c4f2b33728864f080a488db8314e34"
server_key_iv: "4f250f8c528fc8c405891ed4bcf08ab6"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.209638 4341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:07.238766 4341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:07.240566 4341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:07.240728 4341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:07.240970 4341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:07.241281 4341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:07.241333 4341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:07.241367 4341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:07.241395 4341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:07.246994 4341 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:37183
I20260924 13:46:07.247057 4469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:37183 every 8 connection(s)
I20260924 13:46:07.247359 4341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:46:07 dist-test-slave-tfm9 krb5kdc[4234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257567, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:07.255985 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.248042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:53021 (local address 127.19.53.254:35063)
0924 13:46:07.248116 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.248119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.248307 (+ 188us) server_negotiation.cc:407] Connection header received
0924 13:46:07.248768 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.248769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.248783 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.248834 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:07.249714 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.250282 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.251220 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.251416 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.251470 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.253434 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:07.253435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:07.253435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:07.253441 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:07.254892 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:07.255238 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:07.255239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:07.255239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:07.255287 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:07.255492 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:07.255493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:07.255493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:07.255587 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:07.255637 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.255852 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.255921 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 13:46:07.256130 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.248015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:35063 (local address 127.19.53.193:53021)
0924 13:46:07.248186 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:07.248222 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:07.248683 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:07.248891 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:07.248894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:07.249441 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:07.249630 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.249635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.250362 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.250362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.251148 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.251150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.251489 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.251490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.251583 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.252013 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:07.252021 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:07.253353 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:07.254967 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:07.254973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:07.254974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:07.255173 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:07.255347 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:07.255348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:07.255348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:07.255440 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:07.255641 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:07.255645 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:07.255748 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 13:46:07.255825 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 13:46:07.255939 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260924 13:46:07.256702 4470 heartbeater.cc:347] Connected to a master server at 127.19.53.254:35063
I20260924 13:46:07.256807 4470 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:07.256976 4470 heartbeater.cc:511] Master 127.19.53.254:35063 requested a full tablet report, sending...
I20260924 13:46:07.257237 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 4341
I20260924 13:46:07.257346 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:46:07.257503 4279 ts_manager.cc:194] Registered new tserver with Master: 4679b5d6c14142838c3f511134c53cbf (127.19.53.193:37183)
I20260924 13:46:07.257537 19671 external_mini_cluster.cc:1428] Setting key 02eed8991d02ac65222062a7921b641e
I20260924 13:46:07.258297 4279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:53021
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:46:07.280303 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:35063
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:07.361096 4475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:07.361280 4475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:07.361301 4475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:07.362473 4475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:07.362514 4475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:07.362540 4475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:46:07.363946 4475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:35063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:07.364178 4475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:07.364362 4475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:07.369148 4480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:07.369174 4481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:07.369196 4483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:07.369462 4475 server_base.cc:1034] running on GCE node
I20260924 13:46:07.369684 4475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:07.369966 4475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:07.371146 4475 hybrid_clock.cc:648] HybridClock initialized: now 1790257567371133 us; error 27 us; skew 500 ppm
Sep 24 13:46:07 dist-test-slave-tfm9 krb5kdc[4234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257567, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:07.373523 4475 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:46:07.374008 4475 webserver.cc:460] Webserver started at http://127.19.53.194:35025/ using document root <none> and password file <none>
I20260924 13:46:07.374243 4475 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:07.374310 4475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:07.374449 4475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:07.375450 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "9e73a667d20c415a97b81daff7fa6590"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "2f76d25ed892d26c4ae53481fc82ea09"
server_key_iv: "d4b64beea358426c0370a54c0250a9da"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.375787 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "9e73a667d20c415a97b81daff7fa6590"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "2f76d25ed892d26c4ae53481fc82ea09"
server_key_iv: "d4b64beea358426c0370a54c0250a9da"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.377007 4475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:07.377696 4490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:07.377873 4475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:46:07.377988 4475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "9e73a667d20c415a97b81daff7fa6590"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "2f76d25ed892d26c4ae53481fc82ea09"
server_key_iv: "d4b64beea358426c0370a54c0250a9da"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.378077 4475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:07.409912 4475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:07.411966 4475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:07.412132 4475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:07.412405 4475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:07.412803 4475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:07.412868 4475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:07.412912 4475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:07.412940 4475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:07.418875 4475 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:36765
I20260924 13:46:07.418926 4603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:36765 every 8 connection(s)
I20260924 13:46:07.419241 4475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:46:07 dist-test-slave-tfm9 krb5kdc[4234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257567, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:07.424906 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 4475
I20260924 13:46:07.425029 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:46:07.425261 19671 external_mini_cluster.cc:1428] Setting key 055cf874f2b8f84660cf1eabd6a8c023
I20260924 13:46:07.428292 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.419909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:47795 (local address 127.19.53.254:35063)
0924 13:46:07.419966 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.419968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.420176 (+ 208us) server_negotiation.cc:407] Connection header received
0924 13:46:07.420631 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.420631 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.420642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.420687 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:07.421558 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.422082 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.423016 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.423192 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.423241 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.425455 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:07.425456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:07.425456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:07.425464 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:07.426835 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:07.427310 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:07.427311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:07.427311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:07.427379 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:07.427642 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:07.427642 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:07.427643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:07.427756 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:07.427832 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.428107 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.428189 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:46:07.428501 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.419889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:35063 (local address 127.19.53.194:47795)
0924 13:46:07.420059 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:07.420094 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:07.420549 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:07.420749 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:07.420752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:07.421297 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:07.421485 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.421490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.422151 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.422152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.422947 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.422949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.423260 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.423261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.423361 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.423765 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:07.423773 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:07.425340 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:07.426952 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:07.426959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:07.426961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:07.427207 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:07.427448 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:07.427449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:07.427449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:07.427566 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:07.427824 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:07.427830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:07.427971 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 13:46:07.428084 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 13:46:07.428254 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260924 13:46:07.428941 4604 heartbeater.cc:347] Connected to a master server at 127.19.53.254:35063
I20260924 13:46:07.429045 4604 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:07.429210 4604 heartbeater.cc:511] Master 127.19.53.254:35063 requested a full tablet report, sending...
I20260924 13:46:07.429595 4279 ts_manager.cc:194] Registered new tserver with Master: 9e73a667d20c415a97b81daff7fa6590 (127.19.53.194:36765)
I20260924 13:46:07.430168 4279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:47795
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:46:07.452674 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:35063
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:07.532433 4609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:07.532657 4609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:07.532693 4609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:07.533839 4609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:07.533917 4609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:07.534003 4609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:46:07.535442 4609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:35063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:07.535708 4609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:07.535941 4609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:07.540990 4615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:07.540947 4617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:07.540975 4609 server_base.cc:1034] running on GCE node
W20260924 13:46:07.540947 4614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:07.541391 4609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:07.541675 4609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:07.542853 4609 hybrid_clock.cc:648] HybridClock initialized: now 1790257567542815 us; error 34 us; skew 500 ppm
Sep 24 13:46:07 dist-test-slave-tfm9 krb5kdc[4234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257567, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:07.545264 4609 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:46:07.545682 4609 webserver.cc:460] Webserver started at http://127.19.53.195:46669/ using document root <none> and password file <none>
I20260924 13:46:07.546000 4609 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:07.546069 4609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:07.546205 4609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:07.547310 4609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "10dbb73171c040a4b2c20efb1ec50d57"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "f2e973a81331b8bfa7d2cb81fec60eca"
server_key_iv: "fcb2f58cb0269527a87b0a4c75d45b2a"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.547668 4609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "10dbb73171c040a4b2c20efb1ec50d57"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "f2e973a81331b8bfa7d2cb81fec60eca"
server_key_iv: "fcb2f58cb0269527a87b0a4c75d45b2a"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.548944 4609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:07.549631 4624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:07.549803 4609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:46:07.549902 4609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "10dbb73171c040a4b2c20efb1ec50d57"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "f2e973a81331b8bfa7d2cb81fec60eca"
server_key_iv: "fcb2f58cb0269527a87b0a4c75d45b2a"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.550015 4609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:07.577191 4609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:07.579136 4609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:07.579296 4609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:07.579530 4609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:07.579939 4609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:07.579991 4609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:07.580022 4609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:07.580052 4609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:07.585829 4609 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:39159
I20260924 13:46:07.585891 4737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:39159 every 8 connection(s)
I20260924 13:46:07.586279 4609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:46:07.587064 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 4609
I20260924 13:46:07.587160 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:46:07.587349 19671 external_mini_cluster.cc:1428] Setting key d8c35982391b92958df8e1abd4ec24e0
Sep 24 13:46:07 dist-test-slave-tfm9 krb5kdc[4234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257567, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:07.595845 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.587047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:38609 (local address 127.19.53.254:35063)
0924 13:46:07.587119 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.587122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.587375 (+ 253us) server_negotiation.cc:407] Connection header received
0924 13:46:07.588127 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.588128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.588140 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.588191 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:07.589131 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.589636 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.590832 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.591013 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.591066 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.593087 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:07.593088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:07.593089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:07.593096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:07.594529 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:07.594948 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:07.594949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:07.594950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:07.595007 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:07.595225 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:07.595226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:07.595226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:07.595370 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:07.595428 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.595715 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.595778 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:46:07.596016 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.587018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:35063 (local address 127.19.53.195:38609)
0924 13:46:07.587228 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:07.587270 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:07.588006 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:07.588265 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:07.588269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:07.588847 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:07.589052 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.589056 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.589715 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.589715 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.590763 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.590766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.591082 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.591083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.591187 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.591627 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:07.591635 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:07.593003 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:07.594604 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:07.594609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:07.594611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:07.594819 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:07.595074 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:07.595075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:07.595075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:07.595168 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:07.595454 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:07.595459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:07.595575 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 13:46:07.595672 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 13:46:07.595803 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":101,"threads_started":1}
I20260924 13:46:07.596544 4738 heartbeater.cc:347] Connected to a master server at 127.19.53.254:35063
I20260924 13:46:07.596622 4738 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:07.596755 4738 heartbeater.cc:511] Master 127.19.53.254:35063 requested a full tablet report, sending...
I20260924 13:46:07.597079 4279 ts_manager.cc:194] Registered new tserver with Master: 10dbb73171c040a4b2c20efb1ec50d57 (127.19.53.195:39159)
I20260924 13:46:07.597514 4279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:38609
I20260924 13:46:07.598984 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:46:07.606076 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.600299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.19.53.254:35063)
0924 13:46:07.600401 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.600403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.600488 (+ 85us) server_negotiation.cc:407] Connection header received
0924 13:46:07.600562 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.600563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.600573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.600651 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:07.601175 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.601646 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.602452 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.602644 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.602723 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.603777 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:07.603778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:07.603779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:07.603785 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:07.604871 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:07.605217 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:07.605218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:07.605218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:07.605274 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:07.605529 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:07.605529 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:07.605530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:07.605672 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:07.605771 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.605883 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.605983 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 13:46:07.607425 4279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:46:07.608883 4279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:46:07.615226 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.611084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:39159 (local address 127.0.0.1:37256)
0924 13:46:07.611511 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:07.611521 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:07.611574 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:07.611752 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:07.611755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:07.611757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:07.611822 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.611824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.613147 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.613148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.614199 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.614201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.614933 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.614934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.615010 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.615023 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 13:46:07.615101 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 13:46:07.615153 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":79,"threads_started":1}
I20260924 13:46:07.615240 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.611272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.19.53.195:39159)
0924 13:46:07.611466 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.611471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.611583 (+ 112us) server_negotiation.cc:407] Connection header received
0924 13:46:07.611645 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.611647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.611659 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.611724 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:07.611932 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.613034 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.614281 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.614851 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.614923 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.614983 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.615122 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.615161 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":124,"threads_started":1}
I20260924 13:46:07.615692 4749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.611076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:37183 (local address 127.0.0.1:51820)
0924 13:46:07.611406 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:07.611446 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:07.611512 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:07.611865 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:07.611867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:07.611869 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:07.611959 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.611961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.612977 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.612978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.614290 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.614292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.615385 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.615386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.615466 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.615470 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:07.615537 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 13:46:07.615579 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":29,"thread_start_us":50,"threads_started":1}
I20260924 13:46:07.615789 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.611085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:36765 (local address 127.0.0.1:33462)
0924 13:46:07.611962 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:07.611973 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:07.612021 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:07.612227 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:07.612228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:07.612230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:07.612270 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.612272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.613509 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.613510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.614582 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.614585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.615527 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.615528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.615587 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.615592 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:46:07.615657 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 13:46:07.615694 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":91,"threads_started":1}
I20260924 13:46:07.615906 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.611356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33462 (local address 127.19.53.194:36765)
0924 13:46:07.611546 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.611549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.612076 (+ 527us) server_negotiation.cc:407] Connection header received
0924 13:46:07.612124 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.612126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.612141 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.612214 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:07.612351 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.613409 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.614665 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.615405 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.615481 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.615554 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.615685 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.615735 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260924 13:46:07.615988 4752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.611190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.19.53.193:37183)
0924 13:46:07.611676 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.611682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.611697 (+ 15us) server_negotiation.cc:407] Connection header received
0924 13:46:07.611752 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.611754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.611769 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.611841 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:07.612043 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.612883 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.614401 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.615045 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.615109 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.615162 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.615850 (+ 688us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.615905 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":73,"threads_started":1}
I20260924 13:46:07.616307 4672 tablet_service.cc:1467] Processing CreateTablet for tablet ce0ed757ab9e472cbc315f8975ca8450 (DEFAULT_TABLE table=test-table [id=e815aea55ebb4289a09aeb63088e7220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:07.616499 4538 tablet_service.cc:1467] Processing CreateTablet for tablet ce0ed757ab9e472cbc315f8975ca8450 (DEFAULT_TABLE table=test-table [id=e815aea55ebb4289a09aeb63088e7220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:07.616680 4672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0ed757ab9e472cbc315f8975ca8450. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:07.616822 4538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0ed757ab9e472cbc315f8975ca8450. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:07.616847 4404 tablet_service.cc:1467] Processing CreateTablet for tablet ce0ed757ab9e472cbc315f8975ca8450 (DEFAULT_TABLE table=test-table [id=e815aea55ebb4289a09aeb63088e7220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:07.617158 4404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0ed757ab9e472cbc315f8975ca8450. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:07.619711 4755 tablet_bootstrap.cc:492] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57: Bootstrap starting.
I20260924 13:46:07.620167 4756 tablet_bootstrap.cc:492] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590: Bootstrap starting.
I20260924 13:46:07.620376 4757 tablet_bootstrap.cc:492] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf: Bootstrap starting.
I20260924 13:46:07.620481 4755 tablet_bootstrap.cc:654] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:07.620901 4756 tablet_bootstrap.cc:654] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:07.621037 4755 log.cc:824] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:07.621059 4757 tablet_bootstrap.cc:654] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:07.621443 4756 log.cc:824] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:07.621726 4757 log.cc:824] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:07.621860 4755 tablet_bootstrap.cc:492] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57: No bootstrap required, opened a new log
I20260924 13:46:07.621977 4755 ts_tablet_manager.cc:1397] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:46:07.622224 4756 tablet_bootstrap.cc:492] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590: No bootstrap required, opened a new log
I20260924 13:46:07.622323 4756 ts_tablet_manager.cc:1397] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:46:07.622597 4757 tablet_bootstrap.cc:492] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf: No bootstrap required, opened a new log
I20260924 13:46:07.622676 4757 ts_tablet_manager.cc:1397] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:46:07.623965 4757 raft_consensus.cc:348] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4679b5d6c14142838c3f511134c53cbf" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37183 } } peers { permanent_uuid: "9e73a667d20c415a97b81daff7fa6590" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36765 } } peers { permanent_uuid: "10dbb73171c040a4b2c20efb1ec50d57" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39159 } }
I20260924 13:46:07.624131 4757 raft_consensus.cc:374] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:07.624194 4757 raft_consensus.cc:729] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4679b5d6c14142838c3f511134c53cbf, State: Initialized, Role: FOLLOWER
I20260924 13:46:07.624127 4755 raft_consensus.cc:348] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4679b5d6c14142838c3f511134c53cbf" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37183 } } peers { permanent_uuid: "9e73a667d20c415a97b81daff7fa6590" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36765 } } peers { permanent_uuid: "10dbb73171c040a4b2c20efb1ec50d57" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39159 } }
I20260924 13:46:07.624306 4755 raft_consensus.cc:374] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:07.624228 4756 raft_consensus.cc:348] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4679b5d6c14142838c3f511134c53cbf" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37183 } } peers { permanent_uuid: "9e73a667d20c415a97b81daff7fa6590" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36765 } } peers { permanent_uuid: "10dbb73171c040a4b2c20efb1ec50d57" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39159 } }
I20260924 13:46:07.624337 4757 consensus_queue.cc:260] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4679b5d6c14142838c3f511134c53cbf" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37183 } } peers { permanent_uuid: "9e73a667d20c415a97b81daff7fa6590" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36765 } } peers { permanent_uuid: "10dbb73171c040a4b2c20efb1ec50d57" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39159 } }
I20260924 13:46:07.624379 4756 raft_consensus.cc:374] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:07.624423 4756 raft_consensus.cc:729] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e73a667d20c415a97b81daff7fa6590, State: Initialized, Role: FOLLOWER
I20260924 13:46:07.624423 4755 raft_consensus.cc:729] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10dbb73171c040a4b2c20efb1ec50d57, State: Initialized, Role: FOLLOWER
I20260924 13:46:07.624540 4755 consensus_queue.cc:260] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4679b5d6c14142838c3f511134c53cbf" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37183 } } peers { permanent_uuid: "9e73a667d20c415a97b81daff7fa6590" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36765 } } peers { permanent_uuid: "10dbb73171c040a4b2c20efb1ec50d57" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39159 } }
I20260924 13:46:07.624567 4756 consensus_queue.cc:260] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4679b5d6c14142838c3f511134c53cbf" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37183 } } peers { permanent_uuid: "9e73a667d20c415a97b81daff7fa6590" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36765 } } peers { permanent_uuid: "10dbb73171c040a4b2c20efb1ec50d57" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39159 } }
I20260924 13:46:07.624671 4757 ts_tablet_manager.cc:1428] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:46:07.624825 4756 ts_tablet_manager.cc:1428] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:46:07.624883 4604 heartbeater.cc:503] Master 127.19.53.254:35063 was elected leader, sending a full tablet report...
I20260924 13:46:07.624943 4755 ts_tablet_manager.cc:1428] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 13:46:07.624979 4470 heartbeater.cc:503] Master 127.19.53.254:35063 was elected leader, sending a full tablet report...
I20260924 13:46:07.624948 4738 heartbeater.cc:503] Master 127.19.53.254:35063 was elected leader, sending a full tablet report...
W20260924 13:46:07.669770 4605 tablet.cc:2369] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:46:07.681969 4763 raft_consensus.cc:482] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:46:07.682112 4763 raft_consensus.cc:504] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4679b5d6c14142838c3f511134c53cbf" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37183 } } peers { permanent_uuid: "9e73a667d20c415a97b81daff7fa6590" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36765 } } peers { permanent_uuid: "10dbb73171c040a4b2c20efb1ec50d57" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39159 } }
I20260924 13:46:07.682528 4763 leader_election.cc:288] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4679b5d6c14142838c3f511134c53cbf (127.19.53.193:37183), 10dbb73171c040a4b2c20efb1ec50d57 (127.19.53.195:39159)
I20260924 13:46:07.682991 4762 raft_consensus.cc:482] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:46:07.683126 4762 raft_consensus.cc:504] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4679b5d6c14142838c3f511134c53cbf" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37183 } } peers { permanent_uuid: "9e73a667d20c415a97b81daff7fa6590" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36765 } } peers { permanent_uuid: "10dbb73171c040a4b2c20efb1ec50d57" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39159 } }
I20260924 13:46:07.683616 4762 leader_election.cc:288] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4679b5d6c14142838c3f511134c53cbf (127.19.53.193:37183), 9e73a667d20c415a97b81daff7fa6590 (127.19.53.194:36765)
I20260924 13:46:07.686867 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.682896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:49953 (local address 127.19.53.195:39159)
0924 13:46:07.682950 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.682952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.682965 (+ 13us) server_negotiation.cc:407] Connection header received
0924 13:46:07.683068 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.683070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.683079 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.683109 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:07.683328 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.684024 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.685787 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.686448 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.686502 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.686512 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.686760 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.686800 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:46:07.686870 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.682704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:37183 (local address 127.19.53.194:38039)
0924 13:46:07.682757 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:07.682772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:07.682831 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:07.683040 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:07.683041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:07.683043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:07.683130 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.683132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.684155 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.684156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.685654 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.685657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.686560 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.686561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.686652 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.686660 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 13:46:07.686721 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 13:46:07.686786 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":92}
I20260924 13:46:07.687335 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.682710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:39159 (local address 127.19.53.194:49953)
0924 13:46:07.682912 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:07.682922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:07.682979 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:07.683176 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:07.683176 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:07.683178 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:07.683251 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.683253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.684137 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.684138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.685654 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.685657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.686578 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.686579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.686652 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.686660 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 13:46:07.687186 (+ 526us) client_negotiation.cc:240] Negotiation successful
0924 13:46:07.687234 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":159,"thread_start_us":132,"threads_started":1}
I20260924 13:46:07.687480 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.683585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:36765 (local address 127.19.53.195:38645)
0924 13:46:07.683621 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:07.683633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:07.683674 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:07.684066 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:07.684067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:07.684069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:07.684140 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.684142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.685232 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.685233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.686442 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.686445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.687226 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.687227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.687303 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.687307 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:07.687355 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 13:46:07.687390 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 13:46:07.687500 4752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.682744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:38039 (local address 127.19.53.193:37183)
0924 13:46:07.682795 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.682797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.682815 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:46:07.682948 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.682949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.682965 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.683024 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:07.683284 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.684048 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.685788 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.686432 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.686514 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.686523 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.687354 (+ 831us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.687408 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 13:46:07.687619 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.684075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:36743 (local address 127.19.53.193:37183)
0924 13:46:07.684218 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.684219 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.684331 (+ 112us) server_negotiation.cc:407] Connection header received
0924 13:46:07.684390 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.684392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.684402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.684488 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:07.684658 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.685367 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.686628 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.687239 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.687279 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.687287 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.687497 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.687541 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260924 13:46:07.687752 4692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0ed757ab9e472cbc315f8975ca8450" candidate_uuid: "9e73a667d20c415a97b81daff7fa6590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10dbb73171c040a4b2c20efb1ec50d57" is_pre_election: true
I20260924 13:46:07.687898 4692 raft_consensus.cc:2454] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e73a667d20c415a97b81daff7fa6590 in term 0.
I20260924 13:46:07.687870 4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0ed757ab9e472cbc315f8975ca8450" candidate_uuid: "9e73a667d20c415a97b81daff7fa6590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4679b5d6c14142838c3f511134c53cbf" is_pre_election: true
I20260924 13:46:07.688014 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.683767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:38645 (local address 127.19.53.194:36765)
0924 13:46:07.683899 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.683901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.683910 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:46:07.683949 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.683950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.683961 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.684015 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:07.684231 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.685147 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.686535 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.687118 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.687187 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.687194 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.687921 (+ 727us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.687956 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":163,"server-negotiator.queue_time_us":112}
I20260924 13:46:07.688046 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.683900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:37183 (local address 127.19.53.195:36743)
0924 13:46:07.684275 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:07.684284 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:07.684332 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:07.684500 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:07.684500 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:07.684502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:07.684567 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.684569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.685449 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.685450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.686543 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:07.686545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.687352 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.687352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:07.687417 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.687420 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:46:07.687696 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 13:46:07.687745 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":94,"threads_started":1}
I20260924 13:46:07.688031 4424 raft_consensus.cc:2454] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e73a667d20c415a97b81daff7fa6590 in term 0.
I20260924 13:46:07.688226 4558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0ed757ab9e472cbc315f8975ca8450" candidate_uuid: "10dbb73171c040a4b2c20efb1ec50d57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e73a667d20c415a97b81daff7fa6590" is_pre_election: true
I20260924 13:46:07.688274 4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0ed757ab9e472cbc315f8975ca8450" candidate_uuid: "10dbb73171c040a4b2c20efb1ec50d57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4679b5d6c14142838c3f511134c53cbf" is_pre_election: true
I20260924 13:46:07.688323 4558 raft_consensus.cc:2454] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10dbb73171c040a4b2c20efb1ec50d57 in term 0.
I20260924 13:46:07.688297 4492 leader_election.cc:302] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10dbb73171c040a4b2c20efb1ec50d57, 9e73a667d20c415a97b81daff7fa6590; no voters:
I20260924 13:46:07.688340 4424 raft_consensus.cc:2454] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10dbb73171c040a4b2c20efb1ec50d57 in term 0.
I20260924 13:46:07.688496 4628 leader_election.cc:302] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10dbb73171c040a4b2c20efb1ec50d57, 9e73a667d20c415a97b81daff7fa6590; no voters:
I20260924 13:46:07.688516 4763 raft_consensus.cc:2781] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:46:07.688567 4763 raft_consensus.cc:482] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:46:07.688618 4763 raft_consensus.cc:3037] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:07.688678 4762 raft_consensus.cc:2781] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:46:07.688746 4762 raft_consensus.cc:482] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:46:07.688777 4762 raft_consensus.cc:3037] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:07.689386 4763 raft_consensus.cc:504] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4679b5d6c14142838c3f511134c53cbf" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37183 } } peers { permanent_uuid: "9e73a667d20c415a97b81daff7fa6590" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36765 } } peers { permanent_uuid: "10dbb73171c040a4b2c20efb1ec50d57" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39159 } }
I20260924 13:46:07.689527 4763 leader_election.cc:288] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [CANDIDATE]: Term 1 election: Requested vote from peers 4679b5d6c14142838c3f511134c53cbf (127.19.53.193:37183), 10dbb73171c040a4b2c20efb1ec50d57 (127.19.53.195:39159)
I20260924 13:46:07.689534 4762 raft_consensus.cc:504] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4679b5d6c14142838c3f511134c53cbf" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37183 } } peers { permanent_uuid: "9e73a667d20c415a97b81daff7fa6590" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36765 } } peers { permanent_uuid: "10dbb73171c040a4b2c20efb1ec50d57" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39159 } }
I20260924 13:46:07.689654 4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0ed757ab9e472cbc315f8975ca8450" candidate_uuid: "9e73a667d20c415a97b81daff7fa6590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4679b5d6c14142838c3f511134c53cbf"
I20260924 13:46:07.689682 4692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0ed757ab9e472cbc315f8975ca8450" candidate_uuid: "9e73a667d20c415a97b81daff7fa6590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10dbb73171c040a4b2c20efb1ec50d57"
I20260924 13:46:07.689735 4424 raft_consensus.cc:3037] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:07.689790 4692 raft_consensus.cc:2379] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e73a667d20c415a97b81daff7fa6590 in current term 1: Already voted for candidate 10dbb73171c040a4b2c20efb1ec50d57 in this term.
I20260924 13:46:07.689814 4423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0ed757ab9e472cbc315f8975ca8450" candidate_uuid: "10dbb73171c040a4b2c20efb1ec50d57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4679b5d6c14142838c3f511134c53cbf"
I20260924 13:46:07.689833 4558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0ed757ab9e472cbc315f8975ca8450" candidate_uuid: "10dbb73171c040a4b2c20efb1ec50d57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e73a667d20c415a97b81daff7fa6590"
I20260924 13:46:07.689939 4558 raft_consensus.cc:2379] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10dbb73171c040a4b2c20efb1ec50d57 in current term 1: Already voted for candidate 9e73a667d20c415a97b81daff7fa6590 in this term.
I20260924 13:46:07.689697 4762 leader_election.cc:288] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [CANDIDATE]: Term 1 election: Requested vote from peers 4679b5d6c14142838c3f511134c53cbf (127.19.53.193:37183), 9e73a667d20c415a97b81daff7fa6590 (127.19.53.194:36765)
I20260924 13:46:07.690176 4628 leader_election.cc:302] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10dbb73171c040a4b2c20efb1ec50d57; no voters: 4679b5d6c14142838c3f511134c53cbf, 9e73a667d20c415a97b81daff7fa6590
I20260924 13:46:07.690251 4762 raft_consensus.cc:2726] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 13:46:07.690737 4424 raft_consensus.cc:2454] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e73a667d20c415a97b81daff7fa6590 in term 1.
I20260924 13:46:07.690954 4491 leader_election.cc:302] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4679b5d6c14142838c3f511134c53cbf, 9e73a667d20c415a97b81daff7fa6590; no voters: 10dbb73171c040a4b2c20efb1ec50d57
I20260924 13:46:07.691051 4763 raft_consensus.cc:2781] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:07.691192 4763 raft_consensus.cc:686] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 1 LEADER]: Becoming Leader. State: Replica: 9e73a667d20c415a97b81daff7fa6590, State: Running, Role: LEADER
I20260924 13:46:07.691260 4763 consensus_queue.cc:237] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4679b5d6c14142838c3f511134c53cbf" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37183 } } peers { permanent_uuid: "9e73a667d20c415a97b81daff7fa6590" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36765 } } peers { permanent_uuid: "10dbb73171c040a4b2c20efb1ec50d57" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39159 } }
I20260924 13:46:07.692137 4279 catalog_manager.cc:5495] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e73a667d20c415a97b81daff7fa6590 (127.19.53.194). New cstate: current_term: 1 leader_uuid: "9e73a667d20c415a97b81daff7fa6590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4679b5d6c14142838c3f511134c53cbf" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e73a667d20c415a97b81daff7fa6590" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 36765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10dbb73171c040a4b2c20efb1ec50d57" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39159 } health_report { overall_health: UNKNOWN } } }
I20260924 13:46:07.706672 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.704419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.19.53.194:36765)
0924 13:46:07.704479 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.704481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.704488 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:46:07.704521 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.704522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.704538 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.704602 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:07.704702 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.705186 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.705896 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.706153 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.706205 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.706333 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:07.706432 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:07.706495 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.706526 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.706544 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 13:46:07.708482 4424 raft_consensus.cc:1261] T ce0ed757ab9e472cbc315f8975ca8450 P 4679b5d6c14142838c3f511134c53cbf [term 1 FOLLOWER]: Refusing update from remote peer 9e73a667d20c415a97b81daff7fa6590: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:07.708516 4692 raft_consensus.cc:1261] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 1 FOLLOWER]: Refusing update from remote peer 9e73a667d20c415a97b81daff7fa6590: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:07.708843 4767 consensus_queue.cc:1035] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4679b5d6c14142838c3f511134c53cbf" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:07.709038 4763 consensus_queue.cc:1035] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10dbb73171c040a4b2c20efb1ec50d57" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:07.713971 4279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:46:07.714058 4279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:46:07.714820 4279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bd90dd848a2494d8f4d31d21af94d67: Sending DeleteTablet for 3 replicas of tablet ce0ed757ab9e472cbc315f8975ca8450
I20260924 13:46:07.715142 4538 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0ed757ab9e472cbc315f8975ca8450 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:07 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:33462
I20260924 13:46:07.715219 4404 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0ed757ab9e472cbc315f8975ca8450 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:07 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:51820
I20260924 13:46:07.715410 4780 tablet_replica.cc:330] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590: stopping tablet replica
I20260924 13:46:07.715415 4672 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0ed757ab9e472cbc315f8975ca8450 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:07 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:37256
I20260924 13:46:07.715502 4780 raft_consensus.cc:2229] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:46:07.715602 4780 raft_consensus.cc:2258] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:46:07.715612 4782 tablet_replica.cc:330] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57: stopping tablet replica
I20260924 13:46:07.715703 4782 raft_consensus.cc:2229] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:46:07.715823 4782 raft_consensus.cc:2258] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:46:07.716102 4780 ts_tablet_manager.cc:1905] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:46:07.716141 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 4341
I20260924 13:46:07.718387 4780 ts_tablet_manager.cc:1918] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:46:07.718525 4780 log.cc:1196] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/ce0ed757ab9e472cbc315f8975ca8450
I20260924 13:46:07.718854 4780 ts_tablet_manager.cc:1939] T ce0ed757ab9e472cbc315f8975ca8450 P 9e73a667d20c415a97b81daff7fa6590: Deleting consensus metadata
I20260924 13:46:07.719261 4266 catalog_manager.cc:4841] TS 9e73a667d20c415a97b81daff7fa6590 (127.19.53.194:36765): tablet ce0ed757ab9e472cbc315f8975ca8450 (table test-table [id=e815aea55ebb4289a09aeb63088e7220]) successfully deleted
W20260924 13:46:07.721650 4263 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:37183 (error 108)
I20260924 13:46:07.722201 4749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.722066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:37183 (local address 127.0.0.1:51826)
0924 13:46:07.722128 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:07.722170 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 13:46:07.722199 4782 ts_tablet_manager.cc:1905] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:46:07.722206 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 4475
W20260924 13:46:07.723623 4263 catalog_manager.cc:4584] TS 4679b5d6c14142838c3f511134c53cbf (127.19.53.193:37183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce0ed757ab9e472cbc315f8975ca8450: Network error: Client connection negotiation failed: client connection to 127.19.53.193:37183: connect: Connection refused (error 111)
I20260924 13:46:07.724479 4782 ts_tablet_manager.cc:1918] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:46:07.724556 4782 log.cc:1196] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/ce0ed757ab9e472cbc315f8975ca8450
I20260924 13:46:07.724872 4782 ts_tablet_manager.cc:1939] T ce0ed757ab9e472cbc315f8975ca8450 P 10dbb73171c040a4b2c20efb1ec50d57: Deleting consensus metadata
I20260924 13:46:07.725515 4264 catalog_manager.cc:4841] TS 10dbb73171c040a4b2c20efb1ec50d57 (127.19.53.195:39159): tablet ce0ed757ab9e472cbc315f8975ca8450 (table test-table [id=e815aea55ebb4289a09aeb63088e7220]) successfully deleted
I20260924 13:46:07.727388 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 4609
I20260924 13:46:07.732483 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 4247
2026-09-24T13:46:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1962 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T13:46:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:07Z Disabled control of system clock
I20260924 13:46:07.759662 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:40445
--webserver_interface=127.19.53.254
--webserver_port=0
--builtin_ntp_servers=127.19.53.212:33225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:40445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:46:07.844516 4787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:07.844735 4787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:07.844758 4787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:07.845825 4787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:07.845865 4787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:07.845878 4787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:07.845892 4787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:07.845930 4787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:07.847223 4787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:40445
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:40445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:07.847427 4787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:07.847669 4787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:07.852607 4795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:07.852612 4793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:07.852622 4792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:07.852905 4787 server_base.cc:1034] running on GCE node
I20260924 13:46:07.853184 4787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:07.853498 4787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:07.854671 4787 hybrid_clock.cc:648] HybridClock initialized: now 1790257567854632 us; error 51 us; skew 500 ppm
I20260924 13:46:07.855958 4787 webserver.cc:460] Webserver started at http://127.19.53.254:40911/ using document root <none> and password file <none>
I20260924 13:46:07.856240 4787 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:07.856317 4787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:07.856506 4787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:07.857676 4787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "4bfb3ecb218a4434977b3dbeb4643d44"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "4b5d70ba963213e91020f4f3d10eb60e"
server_key_iv: "377f8b3f748c07725c423e0f8fd2cad2"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.858136 4787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "4bfb3ecb218a4434977b3dbeb4643d44"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "4b5d70ba963213e91020f4f3d10eb60e"
server_key_iv: "377f8b3f748c07725c423e0f8fd2cad2"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.859560 4787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:46:07.860306 4801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:07.860495 4787 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:46:07.860611 4787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "4bfb3ecb218a4434977b3dbeb4643d44"
format_stamp: "Formatted at 2026-09-24 13:46:07 on dist-test-slave-tfm9"
server_key: "4b5d70ba963213e91020f4f3d10eb60e"
server_key_iv: "377f8b3f748c07725c423e0f8fd2cad2"
server_key_version: "encryptionkey@0"
I20260924 13:46:07.860692 4787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:07.869436 4787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:07.869735 4787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:07.869900 4787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:07.874244 4787 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:40445
I20260924 13:46:07.874279 4853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:40445 every 8 connection(s)
I20260924 13:46:07.874646 4787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:07.875571 4854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:07.877768 4854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44: Bootstrap starting.
I20260924 13:46:07.878377 4854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:07.878672 4854 log.cc:824] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:07.879391 4854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44: No bootstrap required, opened a new log
I20260924 13:46:07.881001 4854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bfb3ecb218a4434977b3dbeb4643d44" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 40445 } }
I20260924 13:46:07.881134 4854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:07.881156 4854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bfb3ecb218a4434977b3dbeb4643d44, State: Initialized, Role: FOLLOWER
I20260924 13:46:07.881323 4854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bfb3ecb218a4434977b3dbeb4643d44" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 40445 } }
I20260924 13:46:07.881388 4854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:07.881410 4854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:07.881474 4854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:07.882036 4854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bfb3ecb218a4434977b3dbeb4643d44" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 40445 } }
I20260924 13:46:07.882121 4854 leader_election.cc:302] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bfb3ecb218a4434977b3dbeb4643d44; no voters:
I20260924 13:46:07.882331 4854 leader_election.cc:288] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:07.882421 4857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:07.882608 4854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:07.882617 4857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [term 1 LEADER]: Becoming Leader. State: Replica: 4bfb3ecb218a4434977b3dbeb4643d44, State: Running, Role: LEADER
I20260924 13:46:07.882798 4857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bfb3ecb218a4434977b3dbeb4643d44" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 40445 } }
I20260924 13:46:07.883293 4859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bfb3ecb218a4434977b3dbeb4643d44. Latest consensus state: current_term: 1 leader_uuid: "4bfb3ecb218a4434977b3dbeb4643d44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bfb3ecb218a4434977b3dbeb4643d44" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 40445 } } }
I20260924 13:46:07.883383 4859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:07.883388 4858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bfb3ecb218a4434977b3dbeb4643d44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bfb3ecb218a4434977b3dbeb4643d44" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 40445 } } }
I20260924 13:46:07.883630 4866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:07.883751 4858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:07.884332 4866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:07.884374 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 4787
I20260924 13:46:07.884481 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:07.884671 19671 external_mini_cluster.cc:1428] Setting key 61775a90bc1839c33a0aded9fb249c24
I20260924 13:46:07.886034 4866 catalog_manager.cc:1347] Generated new cluster ID: 9ed68842ab3b47dfa3f7d656af45465c
I20260924 13:46:07.886090 4866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:07.890930 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:07.885249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.19.53.254:40445)
0924 13:46:07.885468 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:07.885477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:07.885513 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:46:07.885614 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:07.885619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:07.885670 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:07.885827 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:07.886068 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.887437 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.888680 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:07.889264 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:07.889377 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:07.890228 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:07.890232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:07.890232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:07.890438 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:07.890477 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:07.890490 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:07.890549 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:07.890661 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 13:46:07.890733 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
I20260924 13:46:07.900316 4866 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:07.900808 4866 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:07.904392 4866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44: Generated new TSK 0
I20260924 13:46:07.904561 4866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:07.910319 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:40445
--builtin_ntp_servers=127.19.53.212:33225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:46:08.008577 4878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:08.008765 4878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:08.008787 4878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:08.009981 4878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:08.010021 4878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:08.010046 4878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:46:08.011374 4878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:08.011601 4878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:08.011917 4878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:08.018224 4884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:08.018230 4886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:08.018246 4883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:08.018913 4878 server_base.cc:1034] running on GCE node
I20260924 13:46:08.019138 4878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:08.019387 4878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:08.020576 4878 hybrid_clock.cc:648] HybridClock initialized: now 1790257568020565 us; error 36 us; skew 500 ppm
I20260924 13:46:08.022014 4878 webserver.cc:460] Webserver started at http://127.19.53.193:32845/ using document root <none> and password file <none>
I20260924 13:46:08.022315 4878 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:08.022392 4878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:08.022557 4878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:08.023648 4878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "5a3bf1efb6e44de69ef510f426ac0fd1"
format_stamp: "Formatted at 2026-09-24 13:46:08 on dist-test-slave-tfm9"
server_key: "d314d27dba7fa988ebd8cbfb994a442e"
server_key_iv: "451964d19abf2d5bb6c1de2152429b18"
server_key_version: "encryptionkey@0"
I20260924 13:46:08.024041 4878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "5a3bf1efb6e44de69ef510f426ac0fd1"
format_stamp: "Formatted at 2026-09-24 13:46:08 on dist-test-slave-tfm9"
server_key: "d314d27dba7fa988ebd8cbfb994a442e"
server_key_iv: "451964d19abf2d5bb6c1de2152429b18"
server_key_version: "encryptionkey@0"
I20260924 13:46:08.025484 4878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:08.026502 4892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:08.026685 4878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:46:08.026806 4878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "5a3bf1efb6e44de69ef510f426ac0fd1"
format_stamp: "Formatted at 2026-09-24 13:46:08 on dist-test-slave-tfm9"
server_key: "d314d27dba7fa988ebd8cbfb994a442e"
server_key_iv: "451964d19abf2d5bb6c1de2152429b18"
server_key_version: "encryptionkey@0"
I20260924 13:46:08.026885 4878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:08.043263 4878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:08.043605 4878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:08.043782 4878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:08.044026 4878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:08.044379 4878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:08.044436 4878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:08.044481 4878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:08.044512 4878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:08.050928 4878 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:33883
I20260924 13:46:08.050976 5005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:33883 every 8 connection(s)
I20260924 13:46:08.051337 4878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:46:08.056921 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.052060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:34607 (local address 127.19.53.254:40445)
0924 13:46:08.052112 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:08.052114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:08.052375 (+ 261us) server_negotiation.cc:407] Connection header received
0924 13:46:08.052930 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:08.052931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:08.052948 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:08.053019 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:08.053715 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.054320 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.055346 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.055563 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.055657 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.056599 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:08.056601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:08.056601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:08.056607 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:08.056624 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:08.056628 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:08.056688 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:08.056805 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 13:46:08.056837 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:46:08.057089 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 4878
I20260924 13:46:08.057168 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.052050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:40445 (local address 127.19.53.193:34607)
0924 13:46:08.052243 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:08.052281 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:08.052817 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:08.053096 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:08.053100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:08.053107 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:08.053601 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.053605 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.054425 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.054426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.055228 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.055230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.055669 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.055671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.055813 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.056430 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:46:08.056431 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:08.056460 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:46:08.056483 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:08.056694 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:08.056700 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:08.056703 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:46:08.056798 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 13:46:08.056903 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260924 13:46:08.057199 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:46:08.057392 19671 external_mini_cluster.cc:1428] Setting key f93ef857905583a2c1f2e1d1b3606e04
I20260924 13:46:08.057621 5006 heartbeater.cc:347] Connected to a master server at 127.19.53.254:40445
I20260924 13:46:08.057744 5006 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:08.058002 5006 heartbeater.cc:511] Master 127.19.53.254:40445 requested a full tablet report, sending...
I20260924 13:46:08.058415 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:40445
--builtin_ntp_servers=127.19.53.212:33225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 13:46:08.058589 4818 ts_manager.cc:194] Registered new tserver with Master: 5a3bf1efb6e44de69ef510f426ac0fd1 (127.19.53.193:33883)
I20260924 13:46:08.059191 4818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.53.193:34607
W20260924 13:46:08.139801 5009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:08.139993 5009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:08.140014 5009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:08.141108 5009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:08.141142 5009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:08.141166 5009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:46:08.142503 5009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:08.142683 5009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:08.142913 5009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:08.147773 5017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:08.147887 5009 server_base.cc:1034] running on GCE node
W20260924 13:46:08.147780 5015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:08.147966 5014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:08.148159 5009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:08.148408 5009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:08.149552 5009 hybrid_clock.cc:648] HybridClock initialized: now 1790257568149539 us; error 27 us; skew 500 ppm
I20260924 13:46:08.150768 5009 webserver.cc:460] Webserver started at http://127.19.53.194:41297/ using document root <none> and password file <none>
I20260924 13:46:08.151031 5009 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:08.151075 5009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:08.151244 5009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:08.152267 5009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "7c92132535a54726a1c0ce5d60611f46"
format_stamp: "Formatted at 2026-09-24 13:46:08 on dist-test-slave-tfm9"
server_key: "74e11fcc172e2cf67b3bf06addb298ed"
server_key_iv: "2ed2cbcf66c2db4af0ba81a5cc04b3bf"
server_key_version: "encryptionkey@0"
I20260924 13:46:08.152616 5009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "7c92132535a54726a1c0ce5d60611f46"
format_stamp: "Formatted at 2026-09-24 13:46:08 on dist-test-slave-tfm9"
server_key: "74e11fcc172e2cf67b3bf06addb298ed"
server_key_iv: "2ed2cbcf66c2db4af0ba81a5cc04b3bf"
server_key_version: "encryptionkey@0"
I20260924 13:46:08.153869 5009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:08.154728 5023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:08.154910 5009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:46:08.154979 5009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "7c92132535a54726a1c0ce5d60611f46"
format_stamp: "Formatted at 2026-09-24 13:46:08 on dist-test-slave-tfm9"
server_key: "74e11fcc172e2cf67b3bf06addb298ed"
server_key_iv: "2ed2cbcf66c2db4af0ba81a5cc04b3bf"
server_key_version: "encryptionkey@0"
I20260924 13:46:08.155066 5009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:08.174999 5009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:08.175355 5009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:08.175518 5009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:08.175746 5009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:08.176163 5009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:08.176229 5009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:08.176268 5009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:08.176299 5009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:08.184216 5009 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:37909
I20260924 13:46:08.184285 5136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:37909 every 8 connection(s)
I20260924 13:46:08.184665 5009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:46:08.191682 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.185427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:56675 (local address 127.19.53.254:40445)
0924 13:46:08.185505 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:08.185507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:08.185766 (+ 259us) server_negotiation.cc:407] Connection header received
0924 13:46:08.186656 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:08.186658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:08.186671 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:08.186734 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:08.187742 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.188381 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.189945 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.190273 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.190341 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.191409 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:08.191410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:08.191411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:08.191416 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:08.191431 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:08.191435 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:08.191488 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:08.191567 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 13:46:08.191600 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 13:46:08.191918 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.185399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:40445 (local address 127.19.53.194:56675)
0924 13:46:08.185627 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:08.185671 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:08.186532 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:08.186876 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:08.186881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:08.186892 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:08.187647 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.187653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.188498 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.188499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.189785 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.189790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.190369 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.190371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.190561 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.191244 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:46:08.191244 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:08.191275 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:46:08.191300 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:08.191498 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:08.191503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:08.191506 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:46:08.191573 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 13:46:08.191675 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260924 13:46:08.192370 5137 heartbeater.cc:347] Connected to a master server at 127.19.53.254:40445
I20260924 13:46:08.192487 5137 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:08.192674 5137 heartbeater.cc:511] Master 127.19.53.254:40445 requested a full tablet report, sending...
I20260924 13:46:08.193095 4818 ts_manager.cc:194] Registered new tserver with Master: 7c92132535a54726a1c0ce5d60611f46 (127.19.53.194:37909)
I20260924 13:46:08.193434 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 5009
I20260924 13:46:08.193537 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:46:08.193624 4818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.53.194:56675
I20260924 13:46:08.193781 19671 external_mini_cluster.cc:1428] Setting key 5ecb35e63d0406dc5111da40f798b2c7
I20260924 13:46:08.194738 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:40445
--builtin_ntp_servers=127.19.53.212:33225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:46:08.278843 5140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:08.279042 5140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:08.279062 5140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:08.280145 5140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:08.280184 5140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:08.280208 5140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:46:08.281471 5140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:08.281679 5140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:08.281908 5140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:08.287159 5146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:08.287127 5148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:08.287134 5145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:08.287510 5140 server_base.cc:1034] running on GCE node
I20260924 13:46:08.287729 5140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:08.288014 5140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:08.289167 5140 hybrid_clock.cc:648] HybridClock initialized: now 1790257568289154 us; error 29 us; skew 500 ppm
I20260924 13:46:08.290505 5140 webserver.cc:460] Webserver started at http://127.19.53.195:34847/ using document root <none> and password file <none>
I20260924 13:46:08.290763 5140 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:08.290813 5140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:08.291003 5140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:08.292155 5140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6"
format_stamp: "Formatted at 2026-09-24 13:46:08 on dist-test-slave-tfm9"
server_key: "067778f5b774cc2ebd72f5e1d7ff5521"
server_key_iv: "547de043773bcbaeff033dce518395b9"
server_key_version: "encryptionkey@0"
I20260924 13:46:08.292531 5140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6"
format_stamp: "Formatted at 2026-09-24 13:46:08 on dist-test-slave-tfm9"
server_key: "067778f5b774cc2ebd72f5e1d7ff5521"
server_key_iv: "547de043773bcbaeff033dce518395b9"
server_key_version: "encryptionkey@0"
I20260924 13:46:08.293884 5140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:46:08.294718 5154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:08.294899 5140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:08.295004 5140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6"
format_stamp: "Formatted at 2026-09-24 13:46:08 on dist-test-slave-tfm9"
server_key: "067778f5b774cc2ebd72f5e1d7ff5521"
server_key_iv: "547de043773bcbaeff033dce518395b9"
server_key_version: "encryptionkey@0"
I20260924 13:46:08.295090 5140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:08.305883 5140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:08.306236 5140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:08.306391 5140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:08.306602 5140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:08.306937 5140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:08.306977 5140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:08.307044 5140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:08.307076 5140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:08.313333 5140 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:38431
I20260924 13:46:08.313364 5267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:38431 every 8 connection(s)
I20260924 13:46:08.313694 5140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:46:08.318797 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.314409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:36247 (local address 127.19.53.254:40445)
0924 13:46:08.314458 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:08.314461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:08.314692 (+ 231us) server_negotiation.cc:407] Connection header received
0924 13:46:08.315191 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:08.315192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:08.315201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:08.315253 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:08.315932 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.316426 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.317429 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.317608 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.317655 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.318523 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:08.318524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:08.318525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:08.318529 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:08.318541 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:08.318545 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:08.318590 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:08.318716 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 13:46:08.318740 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:46:08.319032 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.314400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:40445 (local address 127.19.53.195:36247)
0924 13:46:08.314565 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:08.314599 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:08.315082 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:08.315331 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:08.315334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:08.315343 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:08.315860 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.315864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.316507 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.316508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.317358 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.317360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.317679 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.317679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.317787 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.318296 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:46:08.318297 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:08.318415 (+ 118us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:46:08.318438 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:08.318626 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:08.318632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:08.318634 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:46:08.318704 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:46:08.318784 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260924 13:46:08.319602 5268 heartbeater.cc:347] Connected to a master server at 127.19.53.254:40445
I20260924 13:46:08.319697 5268 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:08.319828 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 5140
I20260924 13:46:08.319890 5268 heartbeater.cc:511] Master 127.19.53.254:40445 requested a full tablet report, sending...
I20260924 13:46:08.319942 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:46:08.320138 19671 external_mini_cluster.cc:1428] Setting key 2c5d52df9d5ee6049758dfcbfdd57f0b
I20260924 13:46:08.320288 4818 ts_manager.cc:194] Registered new tserver with Master: 6f2ca4a3cece497aa0b4feba7c6cbdb6 (127.19.53.195:38431)
I20260924 13:46:08.320796 4818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.53.195:36247
I20260924 13:46:08.321077 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:46:08.324761 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.322354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.19.53.254:40445)
0924 13:46:08.322391 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:08.322393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:08.322521 (+ 128us) server_negotiation.cc:407] Connection header received
0924 13:46:08.322552 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:08.322553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:08.322564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:08.322603 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:08.322708 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.323141 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.323806 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.323988 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.324045 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.324588 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:08.324589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:08.324589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:08.324593 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:08.324604 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:08.324607 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:08.324649 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:08.324695 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 13:46:08.324713 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:46:08.326372 4817 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:46:08.327863 4817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:46:08.334153 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.330119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.19.53.195:38431)
0924 13:46:08.330545 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:08.330549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:08.330559 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:46:08.330591 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:08.330593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:08.330606 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:08.330708 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:08.330963 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.331770 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.333124 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.333731 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.333834 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.333942 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:08.334038 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 13:46:08.334073 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":66,"threads_started":1}
I20260924 13:46:08.334143 5276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.330068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:38431 (local address 127.0.0.1:39694)
0924 13:46:08.330242 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:08.330281 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:08.330366 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:08.330721 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:08.330724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:08.330727 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:08.330787 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.330789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.331856 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.331858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.333031 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.333034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.333838 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.333840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.333941 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.333958 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 13:46:08.334027 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 13:46:08.334076 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":59,"threads_started":1}
I20260924 13:46:08.334424 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.330073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:33883 (local address 127.0.0.1:60994)
0924 13:46:08.330365 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:08.330376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:08.330431 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:08.330899 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:08.330899 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:08.330901 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:08.330934 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.330935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.332262 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.332263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.333161 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.333164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.334180 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.334182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.334245 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.334249 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:08.334298 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 13:46:08.334353 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":28,"thread_start_us":87,"threads_started":1}
I20260924 13:46:08.334653 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.330165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.19.53.193:33883)
0924 13:46:08.330325 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:08.330329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:08.330756 (+ 427us) server_negotiation.cc:407] Connection header received
0924 13:46:08.330803 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:08.330806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:08.330822 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:08.330892 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:08.331013 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.332145 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.333250 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.333983 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.334051 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.334144 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:08.334520 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 13:46:08.334562 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260924 13:46:08.334713 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.330265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:37909 (local address 127.0.0.1:38536)
0924 13:46:08.330465 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:08.330473 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:08.330515 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:08.330802 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:08.330803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:08.330805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:08.330857 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.330859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.332258 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.332259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.333383 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.333385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.334496 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.334497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.334553 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.334557 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:08.334606 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 13:46:08.334639 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":25,"thread_start_us":56,"threads_started":1}
I20260924 13:46:08.334913 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.330472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.19.53.194:37909)
0924 13:46:08.330643 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:08.330647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:08.330656 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:46:08.330701 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:08.330702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:08.330714 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:08.330803 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:08.331049 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.332146 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.333467 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.334306 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.334373 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.334462 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:08.334808 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 13:46:08.334846 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260924 13:46:08.334873 5202 tablet_service.cc:1467] Processing CreateTablet for tablet 9665a117a41b459b97d904b50eb09b96 (DEFAULT_TABLE table=test-table [id=0817abdad55f43169054031450f8aaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:08.335258 5202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9665a117a41b459b97d904b50eb09b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:08.337663 5282 tablet_bootstrap.cc:492] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6: Bootstrap starting.
I20260924 13:46:08.338279 5070 tablet_service.cc:1467] Processing CreateTablet for tablet 9665a117a41b459b97d904b50eb09b96 (DEFAULT_TABLE table=test-table [id=0817abdad55f43169054031450f8aaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:08.338403 5282 tablet_bootstrap.cc:654] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:08.338528 5070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9665a117a41b459b97d904b50eb09b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:08.338410 4940 tablet_service.cc:1467] Processing CreateTablet for tablet 9665a117a41b459b97d904b50eb09b96 (DEFAULT_TABLE table=test-table [id=0817abdad55f43169054031450f8aaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:08.338726 5282 log.cc:824] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:08.338744 4940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9665a117a41b459b97d904b50eb09b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:08.339529 5282 tablet_bootstrap.cc:492] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6: No bootstrap required, opened a new log
I20260924 13:46:08.339586 5282 ts_tablet_manager.cc:1397] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:46:08.340983 5284 tablet_bootstrap.cc:492] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46: Bootstrap starting.
I20260924 13:46:08.341522 5284 tablet_bootstrap.cc:654] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:08.341738 5284 log.cc:824] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:08.342077 5286 tablet_bootstrap.cc:492] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1: Bootstrap starting.
I20260924 13:46:08.342453 5284 tablet_bootstrap.cc:492] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46: No bootstrap required, opened a new log
I20260924 13:46:08.342525 5284 ts_tablet_manager.cc:1397] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:46:08.342418 5282 raft_consensus.cc:348] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 38431 } } peers { permanent_uuid: "5a3bf1efb6e44de69ef510f426ac0fd1" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 33883 } } peers { permanent_uuid: "7c92132535a54726a1c0ce5d60611f46" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37909 } }
I20260924 13:46:08.342578 5282 raft_consensus.cc:374] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:08.342622 5282 raft_consensus.cc:729] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2ca4a3cece497aa0b4feba7c6cbdb6, State: Initialized, Role: FOLLOWER
I20260924 13:46:08.342738 5282 consensus_queue.cc:260] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 38431 } } peers { permanent_uuid: "5a3bf1efb6e44de69ef510f426ac0fd1" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 33883 } } peers { permanent_uuid: "7c92132535a54726a1c0ce5d60611f46" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37909 } }
I20260924 13:46:08.343003 5282 ts_tablet_manager.cc:1428] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 13:46:08.343063 5268 heartbeater.cc:503] Master 127.19.53.254:40445 was elected leader, sending a full tablet report...
I20260924 13:46:08.344794 5284 raft_consensus.cc:348] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 38431 } } peers { permanent_uuid: "5a3bf1efb6e44de69ef510f426ac0fd1" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 33883 } } peers { permanent_uuid: "7c92132535a54726a1c0ce5d60611f46" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37909 } }
I20260924 13:46:08.344940 5284 raft_consensus.cc:374] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:08.344985 5284 raft_consensus.cc:729] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c92132535a54726a1c0ce5d60611f46, State: Initialized, Role: FOLLOWER
I20260924 13:46:08.345093 5284 consensus_queue.cc:260] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 38431 } } peers { permanent_uuid: "5a3bf1efb6e44de69ef510f426ac0fd1" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 33883 } } peers { permanent_uuid: "7c92132535a54726a1c0ce5d60611f46" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37909 } }
I20260924 13:46:08.345333 5284 ts_tablet_manager.cc:1428] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:46:08.345434 5137 heartbeater.cc:503] Master 127.19.53.254:40445 was elected leader, sending a full tablet report...
I20260924 13:46:08.346030 5286 tablet_bootstrap.cc:654] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:08.346308 5286 log.cc:824] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:08.346963 5286 tablet_bootstrap.cc:492] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1: No bootstrap required, opened a new log
I20260924 13:46:08.347064 5286 ts_tablet_manager.cc:1397] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260924 13:46:08.348793 5286 raft_consensus.cc:348] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 38431 } } peers { permanent_uuid: "5a3bf1efb6e44de69ef510f426ac0fd1" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 33883 } } peers { permanent_uuid: "7c92132535a54726a1c0ce5d60611f46" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37909 } }
I20260924 13:46:08.348932 5286 raft_consensus.cc:374] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:08.348975 5286 raft_consensus.cc:729] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3bf1efb6e44de69ef510f426ac0fd1, State: Initialized, Role: FOLLOWER
I20260924 13:46:08.349082 5286 consensus_queue.cc:260] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 38431 } } peers { permanent_uuid: "5a3bf1efb6e44de69ef510f426ac0fd1" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 33883 } } peers { permanent_uuid: "7c92132535a54726a1c0ce5d60611f46" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37909 } }
I20260924 13:46:08.349315 5286 ts_tablet_manager.cc:1428] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 13:46:08.349370 5006 heartbeater.cc:503] Master 127.19.53.254:40445 was elected leader, sending a full tablet report...
W20260924 13:46:08.435253 5138 tablet.cc:2369] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:46:08.459673 5287 raft_consensus.cc:482] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:46:08.459797 5287 raft_consensus.cc:504] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 38431 } } peers { permanent_uuid: "5a3bf1efb6e44de69ef510f426ac0fd1" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 33883 } } peers { permanent_uuid: "7c92132535a54726a1c0ce5d60611f46" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37909 } }
I20260924 13:46:08.460111 5287 leader_election.cc:288] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a3bf1efb6e44de69ef510f426ac0fd1 (127.19.53.193:33883), 7c92132535a54726a1c0ce5d60611f46 (127.19.53.194:37909)
I20260924 13:46:08.464181 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.460602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:36681 (local address 127.19.53.194:37909)
0924 13:46:08.460648 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:08.460650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:08.460743 (+ 93us) server_negotiation.cc:407] Connection header received
0924 13:46:08.460773 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:08.460774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:08.460782 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:08.460832 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:08.460994 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.461725 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.463248 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.463816 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.463893 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.463901 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:08.464071 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 13:46:08.464109 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 13:46:08.464181 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.460558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:58717 (local address 127.19.53.193:33883)
0924 13:46:08.460612 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:08.460614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:08.460624 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:46:08.460658 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:08.460659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:08.460672 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:08.460766 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:08.460986 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.461741 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.463248 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.463816 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.463894 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.463901 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:08.464071 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 13:46:08.464109 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 13:46:08.464200 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.460335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:33883 (local address 127.19.53.195:58717)
0924 13:46:08.460393 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:08.460407 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:08.460457 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:08.460795 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:08.460796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:08.460797 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:08.460884 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.460887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.461820 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.461821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.463158 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.463161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.463903 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.463904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.463984 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.463991 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 13:46:08.464051 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 13:46:08.464102 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":74}
I20260924 13:46:08.464211 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.460497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:37909 (local address 127.19.53.195:36681)
0924 13:46:08.460643 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:08.460652 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:08.460708 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:08.460861 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:08.460862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:08.460863 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:08.460920 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.460922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.461820 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.461821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.463158 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:08.463161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.463921 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.463922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:08.463984 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.463991 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 13:46:08.464051 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 13:46:08.464102 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":37,"thread_start_us":91,"threads_started":1}
I20260924 13:46:08.464619 4960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9665a117a41b459b97d904b50eb09b96" candidate_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3bf1efb6e44de69ef510f426ac0fd1" is_pre_election: true
I20260924 13:46:08.464809 4960 raft_consensus.cc:2454] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f2ca4a3cece497aa0b4feba7c6cbdb6 in term 0.
I20260924 13:46:08.464792 5091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9665a117a41b459b97d904b50eb09b96" candidate_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c92132535a54726a1c0ce5d60611f46" is_pre_election: true
I20260924 13:46:08.464931 5091 raft_consensus.cc:2454] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f2ca4a3cece497aa0b4feba7c6cbdb6 in term 0.
I20260924 13:46:08.465049 5155 leader_election.cc:302] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a3bf1efb6e44de69ef510f426ac0fd1, 6f2ca4a3cece497aa0b4feba7c6cbdb6; no voters:
I20260924 13:46:08.465199 5287 raft_consensus.cc:2781] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:46:08.465277 5287 raft_consensus.cc:482] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:46:08.465296 5287 raft_consensus.cc:3037] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:08.466053 5287 raft_consensus.cc:504] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 38431 } } peers { permanent_uuid: "5a3bf1efb6e44de69ef510f426ac0fd1" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 33883 } } peers { permanent_uuid: "7c92132535a54726a1c0ce5d60611f46" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37909 } }
I20260924 13:46:08.466188 5287 leader_election.cc:288] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [CANDIDATE]: Term 1 election: Requested vote from peers 5a3bf1efb6e44de69ef510f426ac0fd1 (127.19.53.193:33883), 7c92132535a54726a1c0ce5d60611f46 (127.19.53.194:37909)
I20260924 13:46:08.466374 4960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9665a117a41b459b97d904b50eb09b96" candidate_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3bf1efb6e44de69ef510f426ac0fd1"
I20260924 13:46:08.466446 4960 raft_consensus.cc:3037] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:08.466441 5091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9665a117a41b459b97d904b50eb09b96" candidate_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c92132535a54726a1c0ce5d60611f46"
I20260924 13:46:08.466523 5091 raft_consensus.cc:3037] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:08.467118 4960 raft_consensus.cc:2454] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f2ca4a3cece497aa0b4feba7c6cbdb6 in term 1.
I20260924 13:46:08.467281 5155 leader_election.cc:302] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a3bf1efb6e44de69ef510f426ac0fd1, 6f2ca4a3cece497aa0b4feba7c6cbdb6; no voters:
I20260924 13:46:08.467401 5287 raft_consensus.cc:2781] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:08.467434 5091 raft_consensus.cc:2454] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f2ca4a3cece497aa0b4feba7c6cbdb6 in term 1.
I20260924 13:46:08.467558 5287 raft_consensus.cc:686] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [term 1 LEADER]: Becoming Leader. State: Replica: 6f2ca4a3cece497aa0b4feba7c6cbdb6, State: Running, Role: LEADER
I20260924 13:46:08.467655 5287 consensus_queue.cc:237] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 38431 } } peers { permanent_uuid: "5a3bf1efb6e44de69ef510f426ac0fd1" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 33883 } } peers { permanent_uuid: "7c92132535a54726a1c0ce5d60611f46" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37909 } }
I20260924 13:46:08.468557 4818 catalog_manager.cc:5495] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f2ca4a3cece497aa0b4feba7c6cbdb6 (127.19.53.195). New cstate: current_term: 1 leader_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2ca4a3cece497aa0b4feba7c6cbdb6" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 38431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a3bf1efb6e44de69ef510f426ac0fd1" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 33883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c92132535a54726a1c0ce5d60611f46" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37909 } health_report { overall_health: UNKNOWN } } }
I20260924 13:46:08.481981 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.479650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39708 (local address 127.19.53.195:38431)
0924 13:46:08.479710 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:08.479712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:08.479721 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:46:08.479751 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:08.479752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:08.479775 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:08.479879 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:08.479985 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.480443 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.481191 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:08.481393 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:08.481464 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:08.481626 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:08.481726 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:08.481820 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:08.481846 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 13:46:08.481863 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:46:08.483670 5091 raft_consensus.cc:1261] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46 [term 1 FOLLOWER]: Refusing update from remote peer 6f2ca4a3cece497aa0b4feba7c6cbdb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:08.483762 4960 raft_consensus.cc:1261] T 9665a117a41b459b97d904b50eb09b96 P 5a3bf1efb6e44de69ef510f426ac0fd1 [term 1 FOLLOWER]: Refusing update from remote peer 6f2ca4a3cece497aa0b4feba7c6cbdb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:08.484004 5292 consensus_queue.cc:1035] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c92132535a54726a1c0ce5d60611f46" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 37909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:08.484127 5287 consensus_queue.cc:1035] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a3bf1efb6e44de69ef510f426ac0fd1" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 33883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:08.486261 5295 mvcc.cc:205] Tried to move back new op lower bound from 7332895000506609664 to 7332895000444317696. Current Snapshot: MvccSnapshot[applied={T|T < 7332895000506609664}]
I20260924 13:46:08.487288 5297 mvcc.cc:205] Tried to move back new op lower bound from 7332895000506609664 to 7332895000444317696. Current Snapshot: MvccSnapshot[applied={T|T < 7332895000506609664 or (T in {7332895000506609664})}]
I20260924 13:46:08.488682 4818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:46:08.488785 4818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:46:08.489502 4818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bfb3ecb218a4434977b3dbeb4643d44: Sending DeleteTablet for 3 replicas of tablet 9665a117a41b459b97d904b50eb09b96
I20260924 13:46:08.489796 5202 tablet_service.cc:1514] Processing DeleteTablet for tablet 9665a117a41b459b97d904b50eb09b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:08 UTC) from {username='slave'} at 127.0.0.1:39694
I20260924 13:46:08.489868 5070 tablet_service.cc:1514] Processing DeleteTablet for tablet 9665a117a41b459b97d904b50eb09b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:08 UTC) from {username='slave'} at 127.0.0.1:38536
I20260924 13:46:08.490079 4940 tablet_service.cc:1514] Processing DeleteTablet for tablet 9665a117a41b459b97d904b50eb09b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:08 UTC) from {username='slave'} at 127.0.0.1:60994
I20260924 13:46:08.490113 5306 tablet_replica.cc:330] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46: stopping tablet replica
I20260924 13:46:08.490208 5306 raft_consensus.cc:2229] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:46:08.490247 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 4878
I20260924 13:46:08.490312 5306 raft_consensus.cc:2258] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:46:08.491010 5306 ts_tablet_manager.cc:1905] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:46:08.493019 5306 ts_tablet_manager.cc:1918] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:46:08.493093 5306 log.cc:1196] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/9665a117a41b459b97d904b50eb09b96
I20260924 13:46:08.493333 5306 ts_tablet_manager.cc:1939] T 9665a117a41b459b97d904b50eb09b96 P 7c92132535a54726a1c0ce5d60611f46: Deleting consensus metadata
I20260924 13:46:08.493726 4802 catalog_manager.cc:4841] TS 7c92132535a54726a1c0ce5d60611f46 (127.19.53.194:37909): tablet 9665a117a41b459b97d904b50eb09b96 (table test-table [id=0817abdad55f43169054031450f8aaf4]) successfully deleted
W20260924 13:46:08.497180 4802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:33883 (error 108)
I20260924 13:46:08.497407 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 5009
I20260924 13:46:08.497550 5305 tablet_replica.cc:330] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6: stopping tablet replica
I20260924 13:46:08.497641 5305 raft_consensus.cc:2229] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:46:08.497702 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:08.497582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:33883 (local address 127.0.0.1:32770)
0924 13:46:08.497625 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:08.497670 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:33883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 13:46:08.497741 5305 raft_consensus.cc:2258] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 13:46:08.497807 4802 catalog_manager.cc:4584] TS 5a3bf1efb6e44de69ef510f426ac0fd1 (127.19.53.193:33883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9665a117a41b459b97d904b50eb09b96: Network error: Client connection negotiation failed: client connection to 127.19.53.193:33883: connect: Connection refused (error 111)
I20260924 13:46:08.498941 5305 ts_tablet_manager.cc:1905] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:46:08.500901 5305 ts_tablet_manager.cc:1918] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:46:08.501021 5305 log.cc:1196] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/9665a117a41b459b97d904b50eb09b96
I20260924 13:46:08.501315 5305 ts_tablet_manager.cc:1939] T 9665a117a41b459b97d904b50eb09b96 P 6f2ca4a3cece497aa0b4feba7c6cbdb6: Deleting consensus metadata
I20260924 13:46:08.502148 4805 catalog_manager.cc:4841] TS 6f2ca4a3cece497aa0b4feba7c6cbdb6 (127.19.53.195:38431): tablet 9665a117a41b459b97d904b50eb09b96 (table test-table [id=0817abdad55f43169054031450f8aaf4]) successfully deleted
I20260924 13:46:08.503484 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 5140
I20260924 13:46:08.509169 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 4787
2026-09-24T13:46:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (775 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:08 dist-test-slave-tfm9 krb5kdc[5311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:08 dist-test-slave-tfm9 krb5kdc[5311](info): set up 2 sockets
Sep 24 13:46:08 dist-test-slave-tfm9 krb5kdc[5311](info): commencing operation
krb5kdc: starting...
W20260924 13:46:09.583211 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:09 dist-test-slave-tfm9 krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:46:09.698545 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33291
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:33291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:09.778718 5324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:09.778927 5324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:09.778949 5324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:09.780098 5324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:09.780138 5324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:09.780151 5324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:09.780164 5324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:09.780175 5324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:09.781559 5324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:33291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:09.781738 5324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:09.781924 5324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:09.786734 5330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:09.786834 5324 server_base.cc:1034] running on GCE node
W20260924 13:46:09.786736 5332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:09.786767 5329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:09.787251 5324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:09.787575 5324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:09.788775 5324 hybrid_clock.cc:648] HybridClock initialized: now 1790257569788760 us; error 48 us; skew 500 ppm
Sep 24 13:46:09 dist-test-slave-tfm9 krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257569, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:09.791218 5324 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:46:09.791699 5324 webserver.cc:460] Webserver started at http://127.19.53.254:40599/ using document root <none> and password file <none>
I20260924 13:46:09.791918 5324 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:09.791986 5324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:09.792132 5324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:09.793180 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "5b4eb143e1c94e3aa6fc794c9c09ce66"
format_stamp: "Formatted at 2026-09-24 13:46:09 on dist-test-slave-tfm9"
server_key: "6719583c34ba34ead4377c433dae137a"
server_key_iv: "f05f3a70894b99b60f46bd4a28d0008e"
server_key_version: "encryptionkey@0"
I20260924 13:46:09.793545 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "5b4eb143e1c94e3aa6fc794c9c09ce66"
format_stamp: "Formatted at 2026-09-24 13:46:09 on dist-test-slave-tfm9"
server_key: "6719583c34ba34ead4377c433dae137a"
server_key_iv: "f05f3a70894b99b60f46bd4a28d0008e"
server_key_version: "encryptionkey@0"
I20260924 13:46:09.794925 5324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:09.795673 5339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:09.795840 5324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:09.795944 5324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "5b4eb143e1c94e3aa6fc794c9c09ce66"
format_stamp: "Formatted at 2026-09-24 13:46:09 on dist-test-slave-tfm9"
server_key: "6719583c34ba34ead4377c433dae137a"
server_key_iv: "f05f3a70894b99b60f46bd4a28d0008e"
server_key_version: "encryptionkey@0"
I20260924 13:46:09.796025 5324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:09.832674 5324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:09.834924 5324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:09.835094 5324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:09.839032 5324 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:33291
I20260924 13:46:09.839076 5391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:33291 every 8 connection(s)
I20260924 13:46:09.839458 5324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:09.840374 5392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:09.842783 5392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66: Bootstrap starting.
I20260924 13:46:09.843164 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 5324
I20260924 13:46:09.843308 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:09.843452 5392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:09.843528 19671 external_mini_cluster.cc:1428] Setting key 4d3372161e901ec0fe1d566917843950
I20260924 13:46:09.843746 5392 log.cc:824] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:09.844606 5392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66: No bootstrap required, opened a new log
I20260924 13:46:09.847280 5392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4eb143e1c94e3aa6fc794c9c09ce66" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33291 } }
I20260924 13:46:09.847457 5392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:09.847502 5392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4eb143e1c94e3aa6fc794c9c09ce66, State: Initialized, Role: FOLLOWER
I20260924 13:46:09.847646 5392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4eb143e1c94e3aa6fc794c9c09ce66" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33291 } }
I20260924 13:46:09.847747 5392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:09.847793 5392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:09.847848 5392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:09.848476 5392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4eb143e1c94e3aa6fc794c9c09ce66" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33291 } }
I20260924 13:46:09.848589 5392 leader_election.cc:302] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b4eb143e1c94e3aa6fc794c9c09ce66; no voters:
I20260924 13:46:09.848754 5392 leader_election.cc:288] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:09.848806 5397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:09.848948 5397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [term 1 LEADER]: Becoming Leader. State: Replica: 5b4eb143e1c94e3aa6fc794c9c09ce66, State: Running, Role: LEADER
I20260924 13:46:09.849012 5392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:09.849033 5397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4eb143e1c94e3aa6fc794c9c09ce66" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33291 } }
I20260924 13:46:09.849380 5398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b4eb143e1c94e3aa6fc794c9c09ce66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4eb143e1c94e3aa6fc794c9c09ce66" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33291 } } }
I20260924 13:46:09.849453 5398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:09.849658 5399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b4eb143e1c94e3aa6fc794c9c09ce66. Latest consensus state: current_term: 1 leader_uuid: "5b4eb143e1c94e3aa6fc794c9c09ce66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4eb143e1c94e3aa6fc794c9c09ce66" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33291 } } }
I20260924 13:46:09.849983 5399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:09.850243 5403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:09.850749 5403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 13:46:09 dist-test-slave-tfm9 krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:09.852360 5403 catalog_manager.cc:1347] Generated new cluster ID: f4db53e50e4f4722ade2c7b633a50230
I20260924 13:46:09.852411 5403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:09.856367 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:09.844083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.19.53.254:33291)
0924 13:46:09.844305 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:09.844314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:09.844353 (+ 39us) server_negotiation.cc:407] Connection header received
0924 13:46:09.844997 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:09.845007 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:09.845075 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:09.845263 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:09.845944 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:09.847433 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:09.848343 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:09.848991 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:09.849511 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:09.852705 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:09.852708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:09.852709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:09.852720 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:09.854721 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:09.855148 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:09.855151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:09.855151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:09.855250 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:09.855474 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:09.855475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:09.855475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:09.855745 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:09.855889 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:09.856033 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 13:46:09.856154 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260924 13:46:09.872124 5403 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:09.873040 5403 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:09.893725 5403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66: Generated new TSK 0
I20260924 13:46:09.893985 5403 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:46:09.927258 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33291
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:10.008380 5418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:10.008603 5418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:10.008642 5418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:10.009770 5418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:10.009845 5418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:10.009891 5418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:46:10.011358 5418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:10.011543 5418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:10.011711 5418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:10.016584 5426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:10.016556 5424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:10.016706 5418 server_base.cc:1034] running on GCE node
W20260924 13:46:10.016570 5423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:10.016985 5418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:10.017236 5418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:10.018399 5418 hybrid_clock.cc:648] HybridClock initialized: now 1790257570018387 us; error 28 us; skew 500 ppm
Sep 24 13:46:10 dist-test-slave-tfm9 krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257570, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:10.020861 5418 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:46:10.021270 5418 webserver.cc:460] Webserver started at http://127.19.53.193:35567/ using document root <none> and password file <none>
I20260924 13:46:10.021493 5418 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:10.021561 5418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:10.021708 5418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:10.022802 5418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "02c3cb01f81046428c0d22f733904338"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "9c2ee1c7bffbcc769f24eb25a74074d0"
server_key_iv: "bbe3bd9a865df95b3147df50793226f1"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.023167 5418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "02c3cb01f81046428c0d22f733904338"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "9c2ee1c7bffbcc769f24eb25a74074d0"
server_key_iv: "bbe3bd9a865df95b3147df50793226f1"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.024489 5418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:46:10.025194 5433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.025357 5418 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:10.025455 5418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "02c3cb01f81046428c0d22f733904338"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "9c2ee1c7bffbcc769f24eb25a74074d0"
server_key_iv: "bbe3bd9a865df95b3147df50793226f1"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.025540 5418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:10.035966 5418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:10.037701 5418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:10.037864 5418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:10.038111 5418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:10.038422 5418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:10.038476 5418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.038517 5418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:10.038544 5418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.044287 5418 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:37833
I20260924 13:46:10.044338 5546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:37833 every 8 connection(s)
I20260924 13:46:10.044636 5418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:46:10 dist-test-slave-tfm9 krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257570, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:10.052320 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 5418
I20260924 13:46:10.052448 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:46:10.052692 19671 external_mini_cluster.cc:1428] Setting key b604cbed95d1e65cb50ec10f8d6a5efa
I20260924 13:46:10.054776 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.045346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:41217 (local address 127.19.53.254:33291)
0924 13:46:10.045440 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:10.045443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:10.045620 (+ 177us) server_negotiation.cc:407] Connection header received
0924 13:46:10.046243 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:10.046244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:10.046257 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:10.046323 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:10.047268 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.047801 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.048790 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.049059 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.049144 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.051459 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:10.051460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:10.051460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:10.051466 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:10.053171 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:10.053619 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:10.053620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:10.053621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:10.053682 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:10.054041 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:10.054043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:10.054043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:10.054204 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:10.054276 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:10.054537 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 13:46:10.054636 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 13:46:10.054807 5549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.045309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33291 (local address 127.19.53.193:41217)
0924 13:46:10.045493 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:10.045538 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:10.046138 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:10.046400 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:10.046403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:10.046964 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:10.047164 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.047169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.047903 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.047904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.048686 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.048688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.049156 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.049157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.049279 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.049798 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:10.049806 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:10.051374 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:10.053283 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:10.053290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:10.053292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:10.053542 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:10.053773 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:10.053774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:10.053774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:10.053887 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:10.054271 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:10.054278 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:10.054396 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 13:46:10.054479 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 13:46:10.054597 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260924 13:46:10.055421 5547 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33291
I20260924 13:46:10.055516 5547 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:10.055706 5547 heartbeater.cc:511] Master 127.19.53.254:33291 requested a full tablet report, sending...
I20260924 13:46:10.056201 5356 ts_manager.cc:194] Registered new tserver with Master: 02c3cb01f81046428c0d22f733904338 (127.19.53.193:37833)
I20260924 13:46:10.056802 5356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:41217
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:46:10.077062 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33291
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:10.157979 5552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:10.158177 5552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:10.158200 5552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:10.159282 5552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:10.159315 5552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:10.159340 5552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:46:10.160677 5552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:10.160849 5552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:10.161090 5552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:10.165923 5557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:10.165984 5560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:10.166000 5558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:10.166154 5552 server_base.cc:1034] running on GCE node
I20260924 13:46:10.166316 5552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:10.166517 5552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:10.167650 5552 hybrid_clock.cc:648] HybridClock initialized: now 1790257570167637 us; error 29 us; skew 500 ppm
Sep 24 13:46:10 dist-test-slave-tfm9 krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257570, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:10.170063 5552 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:46:10.170502 5552 webserver.cc:460] Webserver started at http://127.19.53.194:37645/ using document root <none> and password file <none>
I20260924 13:46:10.170719 5552 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:10.170763 5552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:10.170899 5552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:10.171977 5552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "61c9a8bfb4064b6499e9a1b51bec6a42"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "1c97c417fa17008d2cdcf34e5c32a8f4"
server_key_iv: "402df8d455613816d623c2a1311aee22"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.172329 5552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "61c9a8bfb4064b6499e9a1b51bec6a42"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "1c97c417fa17008d2cdcf34e5c32a8f4"
server_key_iv: "402df8d455613816d623c2a1311aee22"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.173636 5552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:46:10.174434 5567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.174633 5552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.174726 5552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "61c9a8bfb4064b6499e9a1b51bec6a42"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "1c97c417fa17008d2cdcf34e5c32a8f4"
server_key_iv: "402df8d455613816d623c2a1311aee22"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.174794 5552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:10.188109 5552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:10.189973 5552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:10.190135 5552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:10.190382 5552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:10.190693 5552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:10.190725 5552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.190778 5552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:10.190806 5552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.196678 5552 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:39747
I20260924 13:46:10.196715 5680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:39747 every 8 connection(s)
I20260924 13:46:10.197034 5552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:46:10.201623 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 5552
I20260924 13:46:10.201736 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:46:10.201972 19671 external_mini_cluster.cc:1428] Setting key 36bdee3dd03d2aa706f6d964761882de
Sep 24 13:46:10 dist-test-slave-tfm9 krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257570, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:10.206914 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.197758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:39005 (local address 127.19.53.254:33291)
0924 13:46:10.197799 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:10.197801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:10.198101 (+ 300us) server_negotiation.cc:407] Connection header received
0924 13:46:10.198613 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:10.198614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:10.198625 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:10.198687 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:10.199565 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.200128 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.201072 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.201257 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.201310 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.203873 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:10.203874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:10.203874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:10.203881 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:10.205554 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:10.205930 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:10.205931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:10.205931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:10.205995 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:10.206225 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:10.206226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:10.206226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:10.206375 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:10.206450 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:10.206711 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 13:46:10.206805 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:46:10.206931 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.197746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33291 (local address 127.19.53.194:39005)
0924 13:46:10.197976 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:10.198012 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:10.198502 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:10.198742 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:10.198745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:10.199286 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:10.199484 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.199489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.200209 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.200210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.200996 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.200997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.201329 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.201330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.201473 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.202129 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:10.202138 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:10.203778 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:10.205635 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:10.205640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:10.205641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:10.205839 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:10.206062 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:10.206062 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:10.206063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:10.206157 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:10.206451 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:10.206456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:10.206577 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 13:46:10.206662 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 13:46:10.206775 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":102,"threads_started":1}
I20260924 13:46:10.207419 5681 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33291
I20260924 13:46:10.207522 5681 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:10.207724 5681 heartbeater.cc:511] Master 127.19.53.254:33291 requested a full tablet report, sending...
I20260924 13:46:10.208122 5356 ts_manager.cc:194] Registered new tserver with Master: 61c9a8bfb4064b6499e9a1b51bec6a42 (127.19.53.194:39747)
I20260924 13:46:10.208506 5356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:39005
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:46:10.228294 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33291
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:10.307610 5686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:10.307804 5686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:10.307827 5686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:10.308984 5686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:10.309026 5686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:10.309077 5686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:46:10.310546 5686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:33291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:10.310782 5686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:10.311007 5686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:10.315820 5694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:10.315958 5691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:10.315821 5692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:10.315995 5686 server_base.cc:1034] running on GCE node
I20260924 13:46:10.316270 5686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:10.316485 5686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:10.317617 5686 hybrid_clock.cc:648] HybridClock initialized: now 1790257570317601 us; error 41 us; skew 500 ppm
Sep 24 13:46:10 dist-test-slave-tfm9 krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257570, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:10.320024 5686 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:46:10.320518 5686 webserver.cc:460] Webserver started at http://127.19.53.195:41937/ using document root <none> and password file <none>
I20260924 13:46:10.320767 5686 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:10.320834 5686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:10.320989 5686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:10.322058 5686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "48d8175982a54f4bbf2aeea662f11d88"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "a3e42ded724e99faba7817559d6a5f46"
server_key_iv: "a4a5b5c1f51cecbffc4f4b5c3b6af00e"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.322422 5686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "48d8175982a54f4bbf2aeea662f11d88"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "a3e42ded724e99faba7817559d6a5f46"
server_key_iv: "a4a5b5c1f51cecbffc4f4b5c3b6af00e"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.323719 5686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:46:10.324432 5701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.324597 5686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:10.324698 5686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "48d8175982a54f4bbf2aeea662f11d88"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "a3e42ded724e99faba7817559d6a5f46"
server_key_iv: "a4a5b5c1f51cecbffc4f4b5c3b6af00e"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.324772 5686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:10.337182 5686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:10.339136 5686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:10.339298 5686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:10.339517 5686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:10.339850 5686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:10.339905 5686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.339946 5686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:10.339973 5686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.345531 5686 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:40825
I20260924 13:46:10.345604 5814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:40825 every 8 connection(s)
I20260924 13:46:10.345973 5686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:46:10 dist-test-slave-tfm9 krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257570, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:10.353163 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 5686
I20260924 13:46:10.353251 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:46:10.353459 19671 external_mini_cluster.cc:1428] Setting key 89ce07c75864b3d090523d7fb740756c
I20260924 13:46:10.355736 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.346672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:36299 (local address 127.19.53.254:33291)
0924 13:46:10.346715 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:10.346717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:10.346943 (+ 226us) server_negotiation.cc:407] Connection header received
0924 13:46:10.347475 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:10.347476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:10.347488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:10.347543 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:10.348482 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.349042 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.350288 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.350490 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.350588 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.352655 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:10.352656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:10.352657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:10.352663 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:10.354407 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:10.354781 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:10.354782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:10.354782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:10.354857 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:10.355102 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:10.355103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:10.355104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:10.355225 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:10.355300 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:10.355570 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 13:46:10.355645 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:46:10.355934 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.346651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:33291 (local address 127.19.53.195:36299)
0924 13:46:10.346823 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:10.346861 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:10.347371 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:10.347615 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:10.347619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:10.348179 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:10.348374 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.348379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.349132 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.349133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.350192 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.350195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.350601 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.350602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.350718 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.351231 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:10.351238 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:10.352575 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:10.354467 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:10.354473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:10.354476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:10.354691 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:10.354925 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:10.354926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:10.354926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:10.355036 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:10.355291 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:10.355298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:10.355441 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 13:46:10.355550 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 13:46:10.355703 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260924 13:46:10.356395 5815 heartbeater.cc:347] Connected to a master server at 127.19.53.254:33291
I20260924 13:46:10.356493 5815 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:10.356665 5815 heartbeater.cc:511] Master 127.19.53.254:33291 requested a full tablet report, sending...
I20260924 13:46:10.357009 5356 ts_manager.cc:194] Registered new tserver with Master: 48d8175982a54f4bbf2aeea662f11d88 (127.19.53.195:40825)
I20260924 13:46:10.357331 5356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:36299
I20260924 13:46:10.365296 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:46:10.372319 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.366574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.19.53.254:33291)
0924 13:46:10.366630 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:10.366632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:10.366749 (+ 117us) server_negotiation.cc:407] Connection header received
0924 13:46:10.366783 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:10.366784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:10.366792 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:10.366851 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:10.367349 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.367797 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.368455 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.368633 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.368679 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.369833 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:10.369834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:10.369834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:10.369840 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:10.371244 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:10.371604 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:10.371604 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:10.371605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:10.371652 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:10.371839 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:10.371839 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:10.371839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:10.371956 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:10.372038 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:10.372151 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 13:46:10.372198 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 13:46:10.373708 5356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:46:10.375347 5356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:46:10.381645 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.377529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:40825 (local address 127.0.0.1:56408)
0924 13:46:10.378009 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:10.378020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:10.378107 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:10.378494 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:10.378495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:10.378497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:10.378543 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.378545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.379411 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.379413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.380725 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.380729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.381357 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.381357 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.381445 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.381457 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 13:46:10.381528 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 13:46:10.381573 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260924 13:46:10.381771 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.377618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.19.53.195:40825)
0924 13:46:10.377851 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:10.377856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:10.378082 (+ 226us) server_negotiation.cc:407] Connection header received
0924 13:46:10.378209 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:10.378211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:10.378226 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:10.378283 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:10.378654 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.379310 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.380801 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.381264 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.381322 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.381368 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:10.381571 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 13:46:10.381611 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
I20260924 13:46:10.382475 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.377486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:39747 (local address 127.0.0.1:49722)
0924 13:46:10.377634 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:10.377702 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:10.377768 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:10.378197 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:10.378200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:10.378202 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:10.378283 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.378285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.379647 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.379648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.380727 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.380729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.382243 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.382244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.382305 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.382309 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:10.382370 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 13:46:10.382410 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":3,"thread_start_us":53,"threads_started":1}
I20260924 13:46:10.382490 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.377810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49722 (local address 127.19.53.194:39747)
0924 13:46:10.378025 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:10.378029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:10.378040 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:46:10.378081 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:10.378083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:10.378098 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:10.378195 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:10.378359 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.379538 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.381400 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.382137 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.382239 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.382289 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:10.382383 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 13:46:10.382418 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":88,"threads_started":1}
I20260924 13:46:10.382513 5749 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0d1ef9ddcd43838af2d7d3446e60ed (DEFAULT_TABLE table=test-table [id=f4418b71b14049ccabe7e2e6ee5e8cd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:10.382867 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.377696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:37833 (local address 127.0.0.1:38534)
0924 13:46:10.378008 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:10.378020 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:10.378074 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:10.378307 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:10.378307 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:10.378310 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:10.378384 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.378387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.380840 (+ 2453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.380840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.381639 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.381641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.382589 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.382590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.382682 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.382687 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:46:10.382744 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 13:46:10.382782 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":36,"threads_started":1}
I20260924 13:46:10.382903 5749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0d1ef9ddcd43838af2d7d3446e60ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:10.383080 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.377822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.19.53.193:37833)
0924 13:46:10.378046 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:10.378050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:10.378062 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:46:10.378143 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:10.378145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:10.378160 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:10.378229 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:10.378468 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.379527 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.381728 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.382435 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.382503 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.382554 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:10.382955 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 13:46:10.382999 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
I20260924 13:46:10.385375 5829 tablet_bootstrap.cc:492] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88: Bootstrap starting.
I20260924 13:46:10.385696 5615 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0d1ef9ddcd43838af2d7d3446e60ed (DEFAULT_TABLE table=test-table [id=f4418b71b14049ccabe7e2e6ee5e8cd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:10.386073 5615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0d1ef9ddcd43838af2d7d3446e60ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:10.386185 5829 tablet_bootstrap.cc:654] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:10.386687 5829 log.cc:824] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:10.387369 5829 tablet_bootstrap.cc:492] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88: No bootstrap required, opened a new log
I20260924 13:46:10.387422 5829 ts_tablet_manager.cc:1397] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:46:10.388530 5831 tablet_bootstrap.cc:492] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42: Bootstrap starting.
I20260924 13:46:10.388993 5829 raft_consensus.cc:348] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c9a8bfb4064b6499e9a1b51bec6a42" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 39747 } } peers { permanent_uuid: "48d8175982a54f4bbf2aeea662f11d88" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 40825 } } peers { permanent_uuid: "02c3cb01f81046428c0d22f733904338" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37833 } }
I20260924 13:46:10.389183 5829 raft_consensus.cc:374] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:10.389225 5829 raft_consensus.cc:729] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d8175982a54f4bbf2aeea662f11d88, State: Initialized, Role: FOLLOWER
I20260924 13:46:10.389127 5481 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0d1ef9ddcd43838af2d7d3446e60ed (DEFAULT_TABLE table=test-table [id=f4418b71b14049ccabe7e2e6ee5e8cd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:10.389416 5831 tablet_bootstrap.cc:654] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:10.389374 5829 consensus_queue.cc:260] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c9a8bfb4064b6499e9a1b51bec6a42" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 39747 } } peers { permanent_uuid: "48d8175982a54f4bbf2aeea662f11d88" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 40825 } } peers { permanent_uuid: "02c3cb01f81046428c0d22f733904338" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37833 } }
I20260924 13:46:10.389459 5481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0d1ef9ddcd43838af2d7d3446e60ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:10.389686 5829 ts_tablet_manager.cc:1428] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:46:10.389830 5815 heartbeater.cc:503] Master 127.19.53.254:33291 was elected leader, sending a full tablet report...
I20260924 13:46:10.390000 5831 log.cc:824] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:10.390825 5831 tablet_bootstrap.cc:492] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42: No bootstrap required, opened a new log
I20260924 13:46:10.390902 5831 ts_tablet_manager.cc:1397] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:46:10.392237 5831 raft_consensus.cc:348] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c9a8bfb4064b6499e9a1b51bec6a42" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 39747 } } peers { permanent_uuid: "48d8175982a54f4bbf2aeea662f11d88" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 40825 } } peers { permanent_uuid: "02c3cb01f81046428c0d22f733904338" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37833 } }
I20260924 13:46:10.392374 5834 tablet_bootstrap.cc:492] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338: Bootstrap starting.
I20260924 13:46:10.392417 5831 raft_consensus.cc:374] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:10.392453 5831 raft_consensus.cc:729] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c9a8bfb4064b6499e9a1b51bec6a42, State: Initialized, Role: FOLLOWER
I20260924 13:46:10.392562 5831 consensus_queue.cc:260] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c9a8bfb4064b6499e9a1b51bec6a42" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 39747 } } peers { permanent_uuid: "48d8175982a54f4bbf2aeea662f11d88" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 40825 } } peers { permanent_uuid: "02c3cb01f81046428c0d22f733904338" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37833 } }
I20260924 13:46:10.392848 5831 ts_tablet_manager.cc:1428] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:46:10.392885 5681 heartbeater.cc:503] Master 127.19.53.254:33291 was elected leader, sending a full tablet report...
I20260924 13:46:10.393006 5834 tablet_bootstrap.cc:654] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:10.393620 5834 log.cc:824] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:10.394310 5834 tablet_bootstrap.cc:492] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338: No bootstrap required, opened a new log
I20260924 13:46:10.394371 5834 ts_tablet_manager.cc:1397] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:46:10.395663 5834 raft_consensus.cc:348] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c9a8bfb4064b6499e9a1b51bec6a42" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 39747 } } peers { permanent_uuid: "48d8175982a54f4bbf2aeea662f11d88" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 40825 } } peers { permanent_uuid: "02c3cb01f81046428c0d22f733904338" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37833 } }
I20260924 13:46:10.395785 5834 raft_consensus.cc:374] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:10.395829 5834 raft_consensus.cc:729] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c3cb01f81046428c0d22f733904338, State: Initialized, Role: FOLLOWER
I20260924 13:46:10.395942 5834 consensus_queue.cc:260] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c9a8bfb4064b6499e9a1b51bec6a42" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 39747 } } peers { permanent_uuid: "48d8175982a54f4bbf2aeea662f11d88" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 40825 } } peers { permanent_uuid: "02c3cb01f81046428c0d22f733904338" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37833 } }
I20260924 13:46:10.396196 5834 ts_tablet_manager.cc:1428] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:46:10.396238 5547 heartbeater.cc:503] Master 127.19.53.254:33291 was elected leader, sending a full tablet report...
W20260924 13:46:10.447616 5682 tablet.cc:2369] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:46:10.545363 5548 tablet.cc:2369] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:46:10.596510 5816 tablet.cc:2369] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:46:10.599637 5837 raft_consensus.cc:482] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:46:10.599798 5837 raft_consensus.cc:504] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c9a8bfb4064b6499e9a1b51bec6a42" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 39747 } } peers { permanent_uuid: "48d8175982a54f4bbf2aeea662f11d88" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 40825 } } peers { permanent_uuid: "02c3cb01f81046428c0d22f733904338" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37833 } }
I20260924 13:46:10.600111 5837 leader_election.cc:288] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61c9a8bfb4064b6499e9a1b51bec6a42 (127.19.53.194:39747), 48d8175982a54f4bbf2aeea662f11d88 (127.19.53.195:40825)
I20260924 13:46:10.603178 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.600472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:37695 (local address 127.19.53.195:40825)
0924 13:46:10.600522 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:10.600524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:10.600535 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:46:10.600597 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:10.600598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:10.600608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:10.600664 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:10.600856 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.601377 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.602441 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.602822 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.602878 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.602905 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:10.603067 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 13:46:10.603107 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 13:46:10.603205 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.600384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:59957 (local address 127.19.53.194:39747)
0924 13:46:10.600452 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:10.600455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:10.600557 (+ 102us) server_negotiation.cc:407] Connection header received
0924 13:46:10.600653 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:10.600655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:10.600667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:10.600720 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:10.600935 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.601392 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.602448 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.602833 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.602922 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.602931 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:10.603101 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 13:46:10.603144 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:46:10.603219 5839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.600297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:39747 (local address 127.19.53.193:59957)
0924 13:46:10.600464 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:10.600474 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:10.600573 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:10.600738 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:10.600739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:10.600740 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:10.600860 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.600862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.601497 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.601498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.602337 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.602339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.602932 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.602933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.602991 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.602994 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:46:10.603082 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 13:46:10.603120 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":15,"thread_start_us":79,"threads_started":1}
I20260924 13:46:10.603406 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.600297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:40825 (local address 127.19.53.193:37695)
0924 13:46:10.600456 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:10.600467 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:10.600523 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:10.600676 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:10.600676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:10.600679 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:10.600781 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.600784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.601457 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.601458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.602326 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.602328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.602901 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.602902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.602972 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.602976 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:10.603026 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:46:10.603105 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260924 13:46:10.603613 5635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0d1ef9ddcd43838af2d7d3446e60ed" candidate_uuid: "02c3cb01f81046428c0d22f733904338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c9a8bfb4064b6499e9a1b51bec6a42" is_pre_election: true
I20260924 13:46:10.603766 5635 raft_consensus.cc:2454] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c3cb01f81046428c0d22f733904338 in term 0.
I20260924 13:46:10.603785 5769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0d1ef9ddcd43838af2d7d3446e60ed" candidate_uuid: "02c3cb01f81046428c0d22f733904338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d8175982a54f4bbf2aeea662f11d88" is_pre_election: true
I20260924 13:46:10.603910 5769 raft_consensus.cc:2454] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c3cb01f81046428c0d22f733904338 in term 0.
I20260924 13:46:10.604012 5437 leader_election.cc:302] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c3cb01f81046428c0d22f733904338, 61c9a8bfb4064b6499e9a1b51bec6a42; no voters:
I20260924 13:46:10.604175 5837 raft_consensus.cc:2781] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:46:10.604254 5837 raft_consensus.cc:482] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:46:10.604291 5837 raft_consensus.cc:3037] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:10.604993 5837 raft_consensus.cc:504] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c9a8bfb4064b6499e9a1b51bec6a42" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 39747 } } peers { permanent_uuid: "48d8175982a54f4bbf2aeea662f11d88" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 40825 } } peers { permanent_uuid: "02c3cb01f81046428c0d22f733904338" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37833 } }
I20260924 13:46:10.605141 5837 leader_election.cc:288] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [CANDIDATE]: Term 1 election: Requested vote from peers 61c9a8bfb4064b6499e9a1b51bec6a42 (127.19.53.194:39747), 48d8175982a54f4bbf2aeea662f11d88 (127.19.53.195:40825)
I20260924 13:46:10.605290 5635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0d1ef9ddcd43838af2d7d3446e60ed" candidate_uuid: "02c3cb01f81046428c0d22f733904338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c9a8bfb4064b6499e9a1b51bec6a42"
I20260924 13:46:10.605288 5769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0d1ef9ddcd43838af2d7d3446e60ed" candidate_uuid: "02c3cb01f81046428c0d22f733904338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d8175982a54f4bbf2aeea662f11d88"
I20260924 13:46:10.605374 5769 raft_consensus.cc:3037] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:10.605376 5635 raft_consensus.cc:3037] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:10.606287 5635 raft_consensus.cc:2454] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c3cb01f81046428c0d22f733904338 in term 1.
I20260924 13:46:10.606287 5769 raft_consensus.cc:2454] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c3cb01f81046428c0d22f733904338 in term 1.
I20260924 13:46:10.606477 5435 leader_election.cc:302] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c3cb01f81046428c0d22f733904338, 48d8175982a54f4bbf2aeea662f11d88; no voters:
I20260924 13:46:10.606590 5837 raft_consensus.cc:2781] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:10.606725 5837 raft_consensus.cc:686] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [term 1 LEADER]: Becoming Leader. State: Replica: 02c3cb01f81046428c0d22f733904338, State: Running, Role: LEADER
I20260924 13:46:10.606820 5837 consensus_queue.cc:237] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c9a8bfb4064b6499e9a1b51bec6a42" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 39747 } } peers { permanent_uuid: "48d8175982a54f4bbf2aeea662f11d88" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 40825 } } peers { permanent_uuid: "02c3cb01f81046428c0d22f733904338" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37833 } }
I20260924 13:46:10.607623 5355 catalog_manager.cc:5495] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c3cb01f81046428c0d22f733904338 (127.19.53.193). New cstate: current_term: 1 leader_uuid: "02c3cb01f81046428c0d22f733904338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c9a8bfb4064b6499e9a1b51bec6a42" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d8175982a54f4bbf2aeea662f11d88" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 40825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02c3cb01f81046428c0d22f733904338" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 37833 } health_report { overall_health: HEALTHY } } }
I20260924 13:46:10.664951 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.662300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.19.53.193:37833)
0924 13:46:10.662381 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:10.662383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:10.662394 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:46:10.662442 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:10.662443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:10.662481 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:10.662563 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:10.662738 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.663466 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.664274 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.664472 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.664525 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.664651 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:10.664735 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:10.664804 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:10.664883 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 13:46:10.664892 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:46:10.666833 5635 raft_consensus.cc:1261] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42 [term 1 FOLLOWER]: Refusing update from remote peer 02c3cb01f81046428c0d22f733904338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:10.666849 5769 raft_consensus.cc:1261] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88 [term 1 FOLLOWER]: Refusing update from remote peer 02c3cb01f81046428c0d22f733904338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:10.667212 5837 consensus_queue.cc:1035] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c9a8bfb4064b6499e9a1b51bec6a42" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:10.667349 5840 consensus_queue.cc:1035] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 02c3cb01f81046428c0d22f733904338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d8175982a54f4bbf2aeea662f11d88" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 40825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:10.669888 5845 mvcc.cc:205] Tried to move back new op lower bound from 7332895009448648704 to 7332895009206112256. Current Snapshot: MvccSnapshot[applied={T|T < 7332895009448648704}]
I20260924 13:46:10.669587 5843 mvcc.cc:205] Tried to move back new op lower bound from 7332895009448648704 to 7332895009206112256. Current Snapshot: MvccSnapshot[applied={T|T < 7332895009448648704}]
I20260924 13:46:10.670373 5844 mvcc.cc:205] Tried to move back new op lower bound from 7332895009448648704 to 7332895009206112256. Current Snapshot: MvccSnapshot[applied={T|T < 7332895009448648704}]
I20260924 13:46:10.672281 5356 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:46:10.672358 5356 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:46:10.673090 5356 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b4eb143e1c94e3aa6fc794c9c09ce66: Sending DeleteTablet for 3 replicas of tablet 2f0d1ef9ddcd43838af2d7d3446e60ed
I20260924 13:46:10.673429 5749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0d1ef9ddcd43838af2d7d3446e60ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:10 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:56408
I20260924 13:46:10.673450 5615 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0d1ef9ddcd43838af2d7d3446e60ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:10 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:49722
I20260924 13:46:10.673636 5481 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0d1ef9ddcd43838af2d7d3446e60ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:10 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:38534
I20260924 13:46:10.673691 5853 tablet_replica.cc:330] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42: stopping tablet replica
I20260924 13:46:10.673739 5852 tablet_replica.cc:330] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88: stopping tablet replica
I20260924 13:46:10.673782 5853 raft_consensus.cc:2229] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:46:10.673810 5852 raft_consensus.cc:2229] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:46:10.674007 5853 raft_consensus.cc:2258] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:46:10.673903 5852 raft_consensus.cc:2258] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:46:10.674140 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 5418
I20260924 13:46:10.675892 5853 ts_tablet_manager.cc:1905] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:46:10.677664 5853 ts_tablet_manager.cc:1918] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:46:10.677748 5853 log.cc:1196] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/2f0d1ef9ddcd43838af2d7d3446e60ed
I20260924 13:46:10.678022 5853 ts_tablet_manager.cc:1939] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 61c9a8bfb4064b6499e9a1b51bec6a42: Deleting consensus metadata
I20260924 13:46:10.678450 5343 catalog_manager.cc:4841] TS 61c9a8bfb4064b6499e9a1b51bec6a42 (127.19.53.194:39747): tablet 2f0d1ef9ddcd43838af2d7d3446e60ed (table test-table [id=f4418b71b14049ccabe7e2e6ee5e8cd9]) successfully deleted
W20260924 13:46:10.679378 5341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:37833 (error 108)
I20260924 13:46:10.679603 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 5552
I20260924 13:46:10.680289 5852 ts_tablet_manager.cc:1905] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:46:10.681366 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.681241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:37833 (local address 127.0.0.1:38556)
0924 13:46:10.681295 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:10.681337 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 13:46:10.681512 5341 catalog_manager.cc:4584] TS 02c3cb01f81046428c0d22f733904338 (127.19.53.193:37833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f0d1ef9ddcd43838af2d7d3446e60ed: Network error: Client connection negotiation failed: client connection to 127.19.53.193:37833: connect: Connection refused (error 111)
I20260924 13:46:10.682242 5852 ts_tablet_manager.cc:1918] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:46:10.682319 5852 log.cc:1196] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/2f0d1ef9ddcd43838af2d7d3446e60ed
I20260924 13:46:10.682595 5852 ts_tablet_manager.cc:1939] T 2f0d1ef9ddcd43838af2d7d3446e60ed P 48d8175982a54f4bbf2aeea662f11d88: Deleting consensus metadata
I20260924 13:46:10.683220 5341 catalog_manager.cc:4841] TS 48d8175982a54f4bbf2aeea662f11d88 (127.19.53.195:40825): tablet 2f0d1ef9ddcd43838af2d7d3446e60ed (table test-table [id=f4418b71b14049ccabe7e2e6ee5e8cd9]) successfully deleted
I20260924 13:46:10.685073 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 5686
I20260924 13:46:10.690122 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 5324
2026-09-24T13:46:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2181 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T13:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:10Z Disabled control of system clock
I20260924 13:46:10.716905 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:41489
--webserver_interface=127.19.53.254
--webserver_port=0
--builtin_ntp_servers=127.19.53.212:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:41489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:46:10.796931 5859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:10.797185 5859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:10.797221 5859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:10.798451 5859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:10.798516 5859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:10.798535 5859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:10.798556 5859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:10.798584 5859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:10.799935 5859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:41489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:41489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:10.800199 5859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:10.800495 5859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:10.805356 5867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:10.805312 5864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:10.805307 5865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:10.805665 5859 server_base.cc:1034] running on GCE node
I20260924 13:46:10.805955 5859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:10.806260 5859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:10.807416 5859 hybrid_clock.cc:648] HybridClock initialized: now 1790257570807405 us; error 29 us; skew 500 ppm
I20260924 13:46:10.808578 5859 webserver.cc:460] Webserver started at http://127.19.53.254:36023/ using document root <none> and password file <none>
I20260924 13:46:10.808784 5859 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:10.808841 5859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:10.808974 5859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:10.810091 5859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "4862bfad04c04e97975e6c7ff7fd9515"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "ffb1d7b8c6873463ebe57034a5ebc0bb"
server_key_iv: "33cf1164a9181ad942a50def5685b105"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.810417 5859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "4862bfad04c04e97975e6c7ff7fd9515"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "ffb1d7b8c6873463ebe57034a5ebc0bb"
server_key_iv: "33cf1164a9181ad942a50def5685b105"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.811712 5859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:10.812402 5873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.812578 5859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:10.812655 5859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "4862bfad04c04e97975e6c7ff7fd9515"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "ffb1d7b8c6873463ebe57034a5ebc0bb"
server_key_iv: "33cf1164a9181ad942a50def5685b105"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.812726 5859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:10.821362 5859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:10.821638 5859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:10.821743 5859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:10.825464 5859 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:41489
I20260924 13:46:10.825511 5925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:41489 every 8 connection(s)
I20260924 13:46:10.825846 5859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:10.826838 5926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:10.829027 5926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515: Bootstrap starting.
I20260924 13:46:10.829588 5926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:10.829888 5926 log.cc:824] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:10.830605 5926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515: No bootstrap required, opened a new log
I20260924 13:46:10.831049 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 5859
I20260924 13:46:10.831151 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:10.831382 19671 external_mini_cluster.cc:1428] Setting key d59bfd92ecad1e49c1cf5a1e8fc1ea91
I20260924 13:46:10.832888 5926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4862bfad04c04e97975e6c7ff7fd9515" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41489 } }
I20260924 13:46:10.833040 5926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:10.833087 5926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4862bfad04c04e97975e6c7ff7fd9515, State: Initialized, Role: FOLLOWER
I20260924 13:46:10.833236 5926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4862bfad04c04e97975e6c7ff7fd9515" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41489 } }
I20260924 13:46:10.833331 5926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:10.833374 5926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:10.833422 5926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:10.834002 5926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4862bfad04c04e97975e6c7ff7fd9515" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41489 } }
I20260924 13:46:10.834121 5926 leader_election.cc:302] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4862bfad04c04e97975e6c7ff7fd9515; no voters:
I20260924 13:46:10.834308 5926 leader_election.cc:288] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:10.834358 5931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:10.834506 5931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [term 1 LEADER]: Becoming Leader. State: Replica: 4862bfad04c04e97975e6c7ff7fd9515, State: Running, Role: LEADER
I20260924 13:46:10.834569 5926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:10.834635 5931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4862bfad04c04e97975e6c7ff7fd9515" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41489 } }
I20260924 13:46:10.834942 5931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4862bfad04c04e97975e6c7ff7fd9515. Latest consensus state: current_term: 1 leader_uuid: "4862bfad04c04e97975e6c7ff7fd9515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4862bfad04c04e97975e6c7ff7fd9515" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41489 } } }
I20260924 13:46:10.835031 5931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:10.834962 5932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4862bfad04c04e97975e6c7ff7fd9515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4862bfad04c04e97975e6c7ff7fd9515" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41489 } } }
I20260924 13:46:10.835177 5932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515 [sys.catalog]: This master's current role is: LEADER
W20260924 13:46:10.836431 5947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:46:10.836483 5947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:46:10.836522 5938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:10.836649 5938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:10.838124 5938 catalog_manager.cc:1347] Generated new cluster ID: 03ba4bce17934fc399f7af3d1bb45d01
I20260924 13:46:10.838177 5938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:10.842583 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.831872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.19.53.254:41489)
0924 13:46:10.832090 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:10.832099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:10.832132 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:46:10.832557 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:10.832563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:10.832619 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:10.832767 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:10.833002 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.834577 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.835712 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.836336 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.836466 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.841853 (+ 5387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:10.841856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:10.841857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:10.842085 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:10.842123 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:10.842136 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:10.842199 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:10.842308 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 13:46:10.842382 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260924 13:46:10.844161 5938 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:10.844641 5938 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:10.847214 5938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515: Generated new TSK 0
I20260924 13:46:10.847383 5938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:10.853176 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41489
--builtin_ntp_servers=127.19.53.212:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:46:10.935770 5950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:10.936069 5950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:10.936110 5950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:10.937281 5950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:10.937348 5950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:10.937393 5950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:46:10.938831 5950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:10.939092 5950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:10.939411 5950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:10.944315 5955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:10.944502 5950 server_base.cc:1034] running on GCE node
W20260924 13:46:10.944358 5956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:10.944311 5958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:10.944857 5950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:10.945123 5950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:10.946293 5950 hybrid_clock.cc:648] HybridClock initialized: now 1790257570946267 us; error 49 us; skew 500 ppm
I20260924 13:46:10.947674 5950 webserver.cc:460] Webserver started at http://127.19.53.193:43405/ using document root <none> and password file <none>
I20260924 13:46:10.947932 5950 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:10.947986 5950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:10.948098 5950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:10.949409 5950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "2545f91defc2449dbe049db03768ce12"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "bf23a9b30f26671cb76791c50fe1b388"
server_key_iv: "6593dbfa0f1812d9133c6cb1655073a7"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.949779 5950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "2545f91defc2449dbe049db03768ce12"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "bf23a9b30f26671cb76791c50fe1b388"
server_key_iv: "6593dbfa0f1812d9133c6cb1655073a7"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.951364 5950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:46:10.952330 5964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.952525 5950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:46:10.952616 5950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "2545f91defc2449dbe049db03768ce12"
format_stamp: "Formatted at 2026-09-24 13:46:10 on dist-test-slave-tfm9"
server_key: "bf23a9b30f26671cb76791c50fe1b388"
server_key_iv: "6593dbfa0f1812d9133c6cb1655073a7"
server_key_version: "encryptionkey@0"
I20260924 13:46:10.952675 5950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:10.977034 5950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:10.977398 5950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:10.977566 5950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:10.977852 5950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:10.978241 5950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:10.978303 5950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.978343 5950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:10.978392 5950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:10.984552 5950 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:43203
I20260924 13:46:10.984645 6077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:43203 every 8 connection(s)
I20260924 13:46:10.985006 5950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:46:10.988498 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 5950
I20260924 13:46:10.988636 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:46:10.988859 19671 external_mini_cluster.cc:1428] Setting key 95098399250c4d369d4dbbef25cb99a2
I20260924 13:46:10.990047 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41489
--builtin_ntp_servers=127.19.53.212:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 13:46:10.990937 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.985848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:39085 (local address 127.19.53.254:41489)
0924 13:46:10.985950 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:10.985953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:10.986230 (+ 277us) server_negotiation.cc:407] Connection header received
0924 13:46:10.986725 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:10.986727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:10.986737 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:10.986796 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:10.987508 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.988012 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.989266 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.989523 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.989594 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.990613 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:10.990614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:10.990615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:10.990620 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:10.990641 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:10.990646 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:10.990713 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:10.990779 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 13:46:10.990814 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 13:46:10.991048 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:10.985806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:41489 (local address 127.19.53.193:39085)
0924 13:46:10.986074 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:10.986109 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:10.986632 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:10.986872 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:10.986876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:10.986885 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:10.987428 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.987433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.988093 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.988094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.989164 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:10.989167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:10.989620 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:10.989621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:10.989790 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:10.990444 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:46:10.990445 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:10.990482 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:46:10.990504 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:10.990718 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:10.990723 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:10.990726 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:46:10.990777 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 13:46:10.990861 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":136,"threads_started":1}
I20260924 13:46:10.991560 6078 heartbeater.cc:347] Connected to a master server at 127.19.53.254:41489
I20260924 13:46:10.991654 6078 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:10.991835 6078 heartbeater.cc:511] Master 127.19.53.254:41489 requested a full tablet report, sending...
I20260924 13:46:10.992353 5890 ts_manager.cc:194] Registered new tserver with Master: 2545f91defc2449dbe049db03768ce12 (127.19.53.193:43203)
I20260924 13:46:10.992949 5890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.53.193:39085
W20260924 13:46:11.072453 6081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:11.072662 6081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:11.072686 6081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:11.073870 6081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:11.073972 6081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:11.074035 6081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:46:11.075407 6081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:11.075662 6081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:11.075981 6081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:11.081027 6089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:11.081080 6087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:11.081035 6086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:11.081212 6081 server_base.cc:1034] running on GCE node
I20260924 13:46:11.081411 6081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:11.081645 6081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:11.082803 6081 hybrid_clock.cc:648] HybridClock initialized: now 1790257571082801 us; error 48 us; skew 500 ppm
I20260924 13:46:11.083973 6081 webserver.cc:460] Webserver started at http://127.19.53.194:43813/ using document root <none> and password file <none>
I20260924 13:46:11.084231 6081 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:11.084300 6081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:11.084477 6081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:11.085515 6081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "2ec45b7e4659489580e8b4505c170228"
format_stamp: "Formatted at 2026-09-24 13:46:11 on dist-test-slave-tfm9"
server_key: "c16c01a26e7b0da279b6c1cab2e7dc97"
server_key_iv: "40108d7aa05589b6d3fd9b05ede963ad"
server_key_version: "encryptionkey@0"
I20260924 13:46:11.085870 6081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "2ec45b7e4659489580e8b4505c170228"
format_stamp: "Formatted at 2026-09-24 13:46:11 on dist-test-slave-tfm9"
server_key: "c16c01a26e7b0da279b6c1cab2e7dc97"
server_key_iv: "40108d7aa05589b6d3fd9b05ede963ad"
server_key_version: "encryptionkey@0"
I20260924 13:46:11.087186 6081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:11.087945 6095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:11.088137 6081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:11.088231 6081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "2ec45b7e4659489580e8b4505c170228"
format_stamp: "Formatted at 2026-09-24 13:46:11 on dist-test-slave-tfm9"
server_key: "c16c01a26e7b0da279b6c1cab2e7dc97"
server_key_iv: "40108d7aa05589b6d3fd9b05ede963ad"
server_key_version: "encryptionkey@0"
I20260924 13:46:11.088316 6081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:11.114046 6081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:11.114399 6081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:11.114562 6081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:11.114823 6081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:11.115217 6081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:11.115262 6081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:11.115298 6081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:11.115336 6081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:11.121258 6081 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:38777
I20260924 13:46:11.121304 6208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:38777 every 8 connection(s)
I20260924 13:46:11.121627 6081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:46:11.125420 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 6081
I20260924 13:46:11.125504 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:46:11.125721 19671 external_mini_cluster.cc:1428] Setting key eb462b8844512788539cebe098cdf6bd
I20260924 13:46:11.126637 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41489
--builtin_ntp_servers=127.19.53.212:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 13:46:11.127359 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.122335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:37549 (local address 127.19.53.254:41489)
0924 13:46:11.122404 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:11.122407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:11.122646 (+ 239us) server_negotiation.cc:407] Connection header received
0924 13:46:11.123278 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:11.123279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:11.123293 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:11.123364 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:11.124080 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.124604 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.125677 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.126064 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.126165 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.127097 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:11.127098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:11.127098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:11.127103 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:11.127117 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:11.127120 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:11.127174 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:11.127253 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 13:46:11.127282 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:46:11.127578 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.122316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:41489 (local address 127.19.53.194:37549)
0924 13:46:11.122526 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:11.122559 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:11.123132 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:11.123439 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:11.123443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:11.123453 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:11.123967 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.123971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.124705 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.124707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.125583 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.125586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.126175 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.126176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.126361 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.126923 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:46:11.126923 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:11.126952 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:46:11.126973 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:11.127177 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:11.127182 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:11.127185 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:46:11.127255 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:46:11.127361 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260924 13:46:11.128074 6209 heartbeater.cc:347] Connected to a master server at 127.19.53.254:41489
I20260924 13:46:11.128175 6209 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:11.128368 6209 heartbeater.cc:511] Master 127.19.53.254:41489 requested a full tablet report, sending...
I20260924 13:46:11.128686 5890 ts_manager.cc:194] Registered new tserver with Master: 2ec45b7e4659489580e8b4505c170228 (127.19.53.194:38777)
I20260924 13:46:11.129060 5890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.53.194:37549
W20260924 13:46:11.207779 6212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:11.208003 6212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:11.208042 6212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:11.209151 6212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:11.209211 6212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:11.209254 6212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:46:11.210618 6212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:11.210871 6212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:11.211153 6212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:11.216008 6220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:11.216009 6217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:11.216035 6218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:11.216202 6212 server_base.cc:1034] running on GCE node
I20260924 13:46:11.216501 6212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:11.216761 6212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:11.218019 6212 hybrid_clock.cc:648] HybridClock initialized: now 1790257571218003 us; error 27 us; skew 500 ppm
I20260924 13:46:11.219305 6212 webserver.cc:460] Webserver started at http://127.19.53.195:43805/ using document root <none> and password file <none>
I20260924 13:46:11.219569 6212 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:11.219621 6212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:11.219796 6212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:11.220842 6212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "eb4b3063a14b42d48b43e551a28ffdda"
format_stamp: "Formatted at 2026-09-24 13:46:11 on dist-test-slave-tfm9"
server_key: "c9fff34cd9f9226c4f7b710f7e83462b"
server_key_iv: "5e2cdd00122433015847b26d2c59448b"
server_key_version: "encryptionkey@0"
I20260924 13:46:11.221213 6212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "eb4b3063a14b42d48b43e551a28ffdda"
format_stamp: "Formatted at 2026-09-24 13:46:11 on dist-test-slave-tfm9"
server_key: "c9fff34cd9f9226c4f7b710f7e83462b"
server_key_iv: "5e2cdd00122433015847b26d2c59448b"
server_key_version: "encryptionkey@0"
I20260924 13:46:11.222623 6212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:11.223459 6226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:11.223654 6212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:11.223739 6212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "eb4b3063a14b42d48b43e551a28ffdda"
format_stamp: "Formatted at 2026-09-24 13:46:11 on dist-test-slave-tfm9"
server_key: "c9fff34cd9f9226c4f7b710f7e83462b"
server_key_iv: "5e2cdd00122433015847b26d2c59448b"
server_key_version: "encryptionkey@0"
I20260924 13:46:11.223793 6212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:11.252969 6212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:11.253242 6212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:11.253381 6212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:11.253572 6212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:11.253950 6212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:11.253988 6212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:11.254010 6212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:11.254058 6212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:11.259696 6212 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:34533
I20260924 13:46:11.259774 6339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:34533 every 8 connection(s)
I20260924 13:46:11.260097 6212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:46:11.262152 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 6212
I20260924 13:46:11.262305 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:46:11.262564 19671 external_mini_cluster.cc:1428] Setting key e3d5d966f3d3084665515b2554a96c01
I20260924 13:46:11.265552 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.260880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:48447 (local address 127.19.53.254:41489)
0924 13:46:11.260947 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:11.260949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:11.261184 (+ 235us) server_negotiation.cc:407] Connection header received
0924 13:46:11.261723 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:11.261725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:11.261737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:11.261789 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:11.262620 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.263143 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.264138 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.264343 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.264418 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.265270 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:11.265272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:11.265272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:11.265277 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:11.265292 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:11.265296 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:11.265356 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:11.265450 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 13:46:11.265475 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 13:46:11.265802 6342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.260873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:41489 (local address 127.19.53.195:48447)
0924 13:46:11.261065 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:11.261099 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:11.261616 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:11.261887 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:11.261890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:11.261899 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:11.262521 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.262525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.263234 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.263235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.264034 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.264036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.264423 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.264424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.264557 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.265123 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:46:11.265123 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:11.265151 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:46:11.265173 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:11.265359 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:11.265364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:11.265368 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:11.265453 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 13:46:11.265553 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260924 13:46:11.266374 6340 heartbeater.cc:347] Connected to a master server at 127.19.53.254:41489
I20260924 13:46:11.266451 6340 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:11.266645 6340 heartbeater.cc:511] Master 127.19.53.254:41489 requested a full tablet report, sending...
I20260924 13:46:11.267026 5890 ts_manager.cc:194] Registered new tserver with Master: eb4b3063a14b42d48b43e551a28ffdda (127.19.53.195:34533)
I20260924 13:46:11.267385 5890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.53.195:48447
I20260924 13:46:11.274405 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:46:11.278682 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.275805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.19.53.254:41489)
0924 13:46:11.275877 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:11.275878 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:11.275931 (+ 53us) server_negotiation.cc:407] Connection header received
0924 13:46:11.275987 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:11.275988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:11.275996 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:11.276041 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:11.276178 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.276644 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.277405 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.277654 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.277707 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.278374 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:11.278375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:11.278376 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:11.278379 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:11.278392 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:11.278395 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:11.278470 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:11.278556 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 13:46:11.278566 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 13:46:11.280167 5890 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:46:11.281673 5890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:46:11.288115 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.283855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.19.53.195:34533)
0924 13:46:11.284320 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:11.284324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:11.284335 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:46:11.284371 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:11.284373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:11.284388 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:11.284473 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:11.284639 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.285739 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.287051 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.287706 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.287778 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.287847 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:11.288002 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 13:46:11.288038 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":83,"threads_started":1}
I20260924 13:46:11.288151 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.283779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:34533 (local address 127.0.0.1:57118)
0924 13:46:11.283940 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:11.283975 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:11.284039 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:11.284485 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:11.284489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:11.284492 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:11.284553 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.284555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.285838 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.285839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.286961 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.286967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.287789 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.287790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.287881 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.287895 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 13:46:11.287978 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 13:46:11.288047 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":66,"thread_start_us":63,"threads_started":1}
I20260924 13:46:11.288384 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.284042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38982 (local address 127.19.53.194:38777)
0924 13:46:11.284189 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:11.284194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:11.284205 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:46:11.284274 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:11.284277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:11.284293 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:11.284373 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:11.284720 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.285784 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.286893 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.287652 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.287737 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.287830 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:11.288262 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 13:46:11.288309 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260924 13:46:11.288151 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.283814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:38777 (local address 127.0.0.1:38982)
0924 13:46:11.284169 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:11.284175 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:11.284200 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:11.284477 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:11.284600 (+ 123us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:11.284603 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:11.284637 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.284638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.285886 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.285886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.286811 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.286813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.287787 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.287789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.287881 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.287895 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 13:46:11.287978 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 13:46:11.288046 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":4,"thread_start_us":100,"threads_started":1}
I20260924 13:46:11.288867 6274 tablet_service.cc:1467] Processing CreateTablet for tablet 0c59c83c6f954859b14734f97ccbad58 (DEFAULT_TABLE table=test-table [id=c4c2b6854d68448fa98b993d8e2c8d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:11.288995 6143 tablet_service.cc:1467] Processing CreateTablet for tablet 0c59c83c6f954859b14734f97ccbad58 (DEFAULT_TABLE table=test-table [id=c4c2b6854d68448fa98b993d8e2c8d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:11.289229 6274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c59c83c6f954859b14734f97ccbad58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:11.289285 6143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c59c83c6f954859b14734f97ccbad58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:11.289325 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.283812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:43203 (local address 127.0.0.1:49428)
0924 13:46:11.284052 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:11.284061 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:11.284112 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:11.284627 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:11.284627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:11.284630 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:11.284672 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.284674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.285535 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.285536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.286797 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.286800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.289080 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.289081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.289144 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.289148 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:11.289197 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 13:46:11.289238 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
I20260924 13:46:11.289655 6353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.284220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49428 (local address 127.19.53.193:43203)
0924 13:46:11.284405 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:11.284410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:11.284420 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:46:11.284456 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:11.284458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:11.284473 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:11.284541 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:11.284754 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.285449 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.287867 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.288970 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.289465 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.289550 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:11.289570 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 13:46:11.289597 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260924 13:46:11.290218 6012 tablet_service.cc:1467] Processing CreateTablet for tablet 0c59c83c6f954859b14734f97ccbad58 (DEFAULT_TABLE table=test-table [id=c4c2b6854d68448fa98b993d8e2c8d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:11.290511 6012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c59c83c6f954859b14734f97ccbad58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:11.292529 6354 tablet_bootstrap.cc:492] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda: Bootstrap starting.
I20260924 13:46:11.292721 6355 tablet_bootstrap.cc:492] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228: Bootstrap starting.
I20260924 13:46:11.293409 6355 tablet_bootstrap.cc:654] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:11.293599 6356 tablet_bootstrap.cc:492] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12: Bootstrap starting.
I20260924 13:46:11.294227 6356 tablet_bootstrap.cc:654] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:11.294236 6354 tablet_bootstrap.cc:654] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:11.294523 6354 log.cc:824] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:11.294524 6356 log.cc:824] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:11.295068 6355 log.cc:824] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:11.295279 6354 tablet_bootstrap.cc:492] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda: No bootstrap required, opened a new log
I20260924 13:46:11.295352 6354 ts_tablet_manager.cc:1397] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 13:46:11.295550 6356 tablet_bootstrap.cc:492] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12: No bootstrap required, opened a new log
I20260924 13:46:11.295616 6356 ts_tablet_manager.cc:1397] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:46:11.295745 6355 tablet_bootstrap.cc:492] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228: No bootstrap required, opened a new log
I20260924 13:46:11.295835 6355 ts_tablet_manager.cc:1397] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 13:46:11.297755 6355 raft_consensus.cc:348] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4b3063a14b42d48b43e551a28ffdda" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34533 } } peers { permanent_uuid: "2545f91defc2449dbe049db03768ce12" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43203 } } peers { permanent_uuid: "2ec45b7e4659489580e8b4505c170228" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 38777 } }
I20260924 13:46:11.297905 6355 raft_consensus.cc:374] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:11.297974 6355 raft_consensus.cc:729] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec45b7e4659489580e8b4505c170228, State: Initialized, Role: FOLLOWER
I20260924 13:46:11.298089 6355 consensus_queue.cc:260] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4b3063a14b42d48b43e551a28ffdda" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34533 } } peers { permanent_uuid: "2545f91defc2449dbe049db03768ce12" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43203 } } peers { permanent_uuid: "2ec45b7e4659489580e8b4505c170228" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 38777 } }
I20260924 13:46:11.298127 6356 raft_consensus.cc:348] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4b3063a14b42d48b43e551a28ffdda" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34533 } } peers { permanent_uuid: "2545f91defc2449dbe049db03768ce12" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43203 } } peers { permanent_uuid: "2ec45b7e4659489580e8b4505c170228" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 38777 } }
I20260924 13:46:11.298127 6354 raft_consensus.cc:348] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4b3063a14b42d48b43e551a28ffdda" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34533 } } peers { permanent_uuid: "2545f91defc2449dbe049db03768ce12" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43203 } } peers { permanent_uuid: "2ec45b7e4659489580e8b4505c170228" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 38777 } }
I20260924 13:46:11.298283 6356 raft_consensus.cc:374] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:11.298283 6354 raft_consensus.cc:374] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:11.298336 6354 raft_consensus.cc:729] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4b3063a14b42d48b43e551a28ffdda, State: Initialized, Role: FOLLOWER
I20260924 13:46:11.298336 6356 raft_consensus.cc:729] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2545f91defc2449dbe049db03768ce12, State: Initialized, Role: FOLLOWER
I20260924 13:46:11.298370 6355 ts_tablet_manager.cc:1428] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 13:46:11.298448 6354 consensus_queue.cc:260] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4b3063a14b42d48b43e551a28ffdda" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34533 } } peers { permanent_uuid: "2545f91defc2449dbe049db03768ce12" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43203 } } peers { permanent_uuid: "2ec45b7e4659489580e8b4505c170228" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 38777 } }
I20260924 13:46:11.298476 6356 consensus_queue.cc:260] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4b3063a14b42d48b43e551a28ffdda" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34533 } } peers { permanent_uuid: "2545f91defc2449dbe049db03768ce12" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43203 } } peers { permanent_uuid: "2ec45b7e4659489580e8b4505c170228" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 38777 } }
I20260924 13:46:11.298519 6209 heartbeater.cc:503] Master 127.19.53.254:41489 was elected leader, sending a full tablet report...
I20260924 13:46:11.298695 6356 ts_tablet_manager.cc:1428] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:46:11.298695 6354 ts_tablet_manager.cc:1428] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 13:46:11.298825 6340 heartbeater.cc:503] Master 127.19.53.254:41489 was elected leader, sending a full tablet report...
I20260924 13:46:11.298862 6078 heartbeater.cc:503] Master 127.19.53.254:41489 was elected leader, sending a full tablet report...
W20260924 13:46:11.372151 6210 tablet.cc:2369] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:46:11.390980 6361 raft_consensus.cc:482] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:46:11.391196 6361 raft_consensus.cc:504] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4b3063a14b42d48b43e551a28ffdda" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34533 } } peers { permanent_uuid: "2545f91defc2449dbe049db03768ce12" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43203 } } peers { permanent_uuid: "2ec45b7e4659489580e8b4505c170228" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 38777 } }
I20260924 13:46:11.391621 6361 leader_election.cc:288] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2545f91defc2449dbe049db03768ce12 (127.19.53.193:43203), 2ec45b7e4659489580e8b4505c170228 (127.19.53.194:38777)
I20260924 13:46:11.394459 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.391792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:38777 (local address 127.19.53.195:48593)
0924 13:46:11.391970 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:11.391980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:11.392044 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:11.392189 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:11.392190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:11.392209 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:11.392280 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.392282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.392910 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.392911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.393650 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.393651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.394207 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.394208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.394276 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.394280 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:11.394335 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 13:46:11.394375 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260924 13:46:11.394520 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.391941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:48593 (local address 127.19.53.194:38777)
0924 13:46:11.391983 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:11.391985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:11.392083 (+ 98us) server_negotiation.cc:407] Connection header received
0924 13:46:11.392111 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:11.392111 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:11.392124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:11.392176 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:11.392357 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.392791 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.393772 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.394132 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.394192 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.394199 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:11.394386 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 13:46:11.394432 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:46:11.394675 6342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.391779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:43203 (local address 127.19.53.195:60911)
0924 13:46:11.391858 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:11.391871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:11.391919 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:11.392078 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:11.392079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:11.392081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:11.392161 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.392163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.392860 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.392861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.393642 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:11.393644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.394390 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.394391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:11.394462 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.394465 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:46:11.394527 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 13:46:11.394593 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":37}
I20260924 13:46:11.394786 6353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.391844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:60911 (local address 127.19.53.193:43203)
0924 13:46:11.391893 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:11.391895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:11.391907 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:46:11.391984 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:11.391985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:11.391999 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:11.392056 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:11.392242 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.392763 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.393741 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.394250 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.394328 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.394337 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:11.394669 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 13:46:11.394714 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 13:46:11.394950 6163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c59c83c6f954859b14734f97ccbad58" candidate_uuid: "eb4b3063a14b42d48b43e551a28ffdda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ec45b7e4659489580e8b4505c170228" is_pre_election: true
I20260924 13:46:11.395120 6163 raft_consensus.cc:2454] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4b3063a14b42d48b43e551a28ffdda in term 0.
I20260924 13:46:11.395084 6032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c59c83c6f954859b14734f97ccbad58" candidate_uuid: "eb4b3063a14b42d48b43e551a28ffdda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2545f91defc2449dbe049db03768ce12" is_pre_election: true
I20260924 13:46:11.395252 6032 raft_consensus.cc:2454] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4b3063a14b42d48b43e551a28ffdda in term 0.
I20260924 13:46:11.395378 6228 leader_election.cc:302] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ec45b7e4659489580e8b4505c170228, eb4b3063a14b42d48b43e551a28ffdda; no voters:
I20260924 13:46:11.395493 6361 raft_consensus.cc:2781] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:46:11.395543 6361 raft_consensus.cc:482] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:46:11.395568 6361 raft_consensus.cc:3037] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:11.396262 6361 raft_consensus.cc:504] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4b3063a14b42d48b43e551a28ffdda" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34533 } } peers { permanent_uuid: "2545f91defc2449dbe049db03768ce12" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43203 } } peers { permanent_uuid: "2ec45b7e4659489580e8b4505c170228" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 38777 } }
I20260924 13:46:11.396399 6361 leader_election.cc:288] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [CANDIDATE]: Term 1 election: Requested vote from peers 2545f91defc2449dbe049db03768ce12 (127.19.53.193:43203), 2ec45b7e4659489580e8b4505c170228 (127.19.53.194:38777)
I20260924 13:46:11.396520 6032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c59c83c6f954859b14734f97ccbad58" candidate_uuid: "eb4b3063a14b42d48b43e551a28ffdda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2545f91defc2449dbe049db03768ce12"
I20260924 13:46:11.396598 6032 raft_consensus.cc:3037] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:11.397315 6032 raft_consensus.cc:2454] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb4b3063a14b42d48b43e551a28ffdda in term 1.
I20260924 13:46:11.397441 6163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c59c83c6f954859b14734f97ccbad58" candidate_uuid: "eb4b3063a14b42d48b43e551a28ffdda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ec45b7e4659489580e8b4505c170228"
I20260924 13:46:11.397495 6230 leader_election.cc:302] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2545f91defc2449dbe049db03768ce12, eb4b3063a14b42d48b43e551a28ffdda; no voters:
I20260924 13:46:11.397523 6163 raft_consensus.cc:3037] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:11.397580 6361 raft_consensus.cc:2781] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:11.397728 6361 raft_consensus.cc:686] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [term 1 LEADER]: Becoming Leader. State: Replica: eb4b3063a14b42d48b43e551a28ffdda, State: Running, Role: LEADER
I20260924 13:46:11.397822 6361 consensus_queue.cc:237] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4b3063a14b42d48b43e551a28ffdda" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34533 } } peers { permanent_uuid: "2545f91defc2449dbe049db03768ce12" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43203 } } peers { permanent_uuid: "2ec45b7e4659489580e8b4505c170228" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 38777 } }
I20260924 13:46:11.398468 6163 raft_consensus.cc:2454] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb4b3063a14b42d48b43e551a28ffdda in term 1.
I20260924 13:46:11.398599 5888 catalog_manager.cc:5495] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda reported cstate change: term changed from 0 to 1, leader changed from <none> to eb4b3063a14b42d48b43e551a28ffdda (127.19.53.195). New cstate: current_term: 1 leader_uuid: "eb4b3063a14b42d48b43e551a28ffdda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4b3063a14b42d48b43e551a28ffdda" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 34533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2545f91defc2449dbe049db03768ce12" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ec45b7e4659489580e8b4505c170228" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 38777 } health_report { overall_health: UNKNOWN } } }
I20260924 13:46:11.403820 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.401678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.19.53.195:34533)
0924 13:46:11.401722 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:11.401723 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:11.401730 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:46:11.401813 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:11.401814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:11.401828 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:11.401892 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:11.402033 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.402458 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.403122 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:11.403289 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:11.403340 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:11.403500 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:11.403576 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:11.403643 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:11.403692 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 13:46:11.403698 (+ 6us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:46:11.405380 6032 raft_consensus.cc:1261] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12 [term 1 FOLLOWER]: Refusing update from remote peer eb4b3063a14b42d48b43e551a28ffdda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:11.405404 6163 raft_consensus.cc:1261] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228 [term 1 FOLLOWER]: Refusing update from remote peer eb4b3063a14b42d48b43e551a28ffdda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:11.405696 6364 consensus_queue.cc:1035] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [LEADER]: Connected to new peer: Peer: permanent_uuid: "2545f91defc2449dbe049db03768ce12" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:11.405807 6361 consensus_queue.cc:1035] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ec45b7e4659489580e8b4505c170228" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 38777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:11.408569 6367 mvcc.cc:205] Tried to move back new op lower bound from 7332895012474368000 to 7332895012446228480. Current Snapshot: MvccSnapshot[applied={T|T < 7332895012474368000 or (T in {7332895012474368000})}]
I20260924 13:46:11.408684 6368 mvcc.cc:205] Tried to move back new op lower bound from 7332895012474368000 to 7332895012446228480. Current Snapshot: MvccSnapshot[applied={T|T < 7332895012474368000 or (T in {7332895012474368000})}]
I20260924 13:46:11.409101 6369 mvcc.cc:205] Tried to move back new op lower bound from 7332895012474368000 to 7332895012446228480. Current Snapshot: MvccSnapshot[applied={T|T < 7332895012474368000}]
I20260924 13:46:11.410092 5888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:46:11.410176 5888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:46:11.410882 5888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4862bfad04c04e97975e6c7ff7fd9515: Sending DeleteTablet for 3 replicas of tablet 0c59c83c6f954859b14734f97ccbad58
I20260924 13:46:11.411177 6274 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c59c83c6f954859b14734f97ccbad58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:11 UTC) from {username='slave'} at 127.0.0.1:57118
I20260924 13:46:11.411216 6143 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c59c83c6f954859b14734f97ccbad58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:11 UTC) from {username='slave'} at 127.0.0.1:38982
I20260924 13:46:11.411231 6012 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c59c83c6f954859b14734f97ccbad58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:11 UTC) from {username='slave'} at 127.0.0.1:49428
I20260924 13:46:11.411487 6378 tablet_replica.cc:330] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12: stopping tablet replica
I20260924 13:46:11.411592 6378 raft_consensus.cc:2229] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:46:11.411693 6378 raft_consensus.cc:2258] T 0c59c83c6f954859b14734f97ccbad58 P 2545f91defc2449dbe049db03768ce12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:46:11.411712 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 5950
I20260924 13:46:11.412398 6377 tablet_replica.cc:330] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228: stopping tablet replica
I20260924 13:46:11.412655 6377 raft_consensus.cc:2229] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:46:11.412751 6377 raft_consensus.cc:2258] T 0c59c83c6f954859b14734f97ccbad58 P 2ec45b7e4659489580e8b4505c170228 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:46:11.413643 6376 tablet_replica.cc:330] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda: stopping tablet replica
I20260924 13:46:11.413707 6376 raft_consensus.cc:2229] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:46:11.413763 6376 raft_consensus.cc:2258] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:46:11.414189 6376 ts_tablet_manager.cc:1905] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:46:11.416214 6376 ts_tablet_manager.cc:1918] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:46:11.416288 6376 log.cc:1196] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/0c59c83c6f954859b14734f97ccbad58
I20260924 13:46:11.416534 6376 ts_tablet_manager.cc:1939] T 0c59c83c6f954859b14734f97ccbad58 P eb4b3063a14b42d48b43e551a28ffdda: Deleting consensus metadata
I20260924 13:46:11.417176 5874 catalog_manager.cc:4841] TS eb4b3063a14b42d48b43e551a28ffdda (127.19.53.195:34533): tablet 0c59c83c6f954859b14734f97ccbad58 (table test-table [id=c4c2b6854d68448fa98b993d8e2c8d6d]) successfully deleted
W20260924 13:46:11.417961 5877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:43203 (error 108)
I20260924 13:46:11.418433 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.418292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:43203 (local address 127.0.0.1:49436)
0924 13:46:11.418358 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:11.418403 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:43203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 13:46:11.418512 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 6081
W20260924 13:46:11.418545 5877 catalog_manager.cc:4584] TS 2545f91defc2449dbe049db03768ce12 (127.19.53.193:43203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c59c83c6f954859b14734f97ccbad58: Network error: Client connection negotiation failed: client connection to 127.19.53.193:43203: connect: Connection refused (error 111)
I20260924 13:46:11.423429 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 6212
I20260924 13:46:11.423574 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:11.423489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:38777 (local address 127.0.0.1:38996)
0924 13:46:11.423529 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:11.423550 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.194:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 13:46:11.428766 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 5859
2026-09-24T13:46:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (738 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:11 dist-test-slave-tfm9 krb5kdc[6382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:11 dist-test-slave-tfm9 krb5kdc[6382](info): set up 2 sockets
Sep 24 13:46:11 dist-test-slave-tfm9 krb5kdc[6382](info): commencing operation
krb5kdc: starting...
W20260924 13:46:12.467343 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:12 dist-test-slave-tfm9 krb5kdc[6382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:46:12.582703 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:38837
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:38837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:12.662619 6395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:12.662815 6395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:12.662837 6395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:12.663902 6395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:12.663934 6395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:12.663947 6395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:12.663960 6395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:12.663972 6395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:12.665352 6395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:38837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:38837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:12.665539 6395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:12.665695 6395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:12.670665 6401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:12.670691 6400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:12.670662 6403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:12.670763 6395 server_base.cc:1034] running on GCE node
I20260924 13:46:12.671286 6395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:12.671591 6395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:12.672737 6395 hybrid_clock.cc:648] HybridClock initialized: now 1790257572672728 us; error 50 us; skew 500 ppm
Sep 24 13:46:12 dist-test-slave-tfm9 krb5kdc[6382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257572, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:12.675227 6395 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:46:12.675625 6395 webserver.cc:460] Webserver started at http://127.19.53.254:44519/ using document root <none> and password file <none>
I20260924 13:46:12.675846 6395 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:12.675913 6395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:12.676059 6395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:12.677069 6395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "d295dd19b0604e7a8c182ee8b768cf13"
format_stamp: "Formatted at 2026-09-24 13:46:12 on dist-test-slave-tfm9"
server_key: "340e3f01ed0d409eba881ad957b9c313"
server_key_iv: "1be92201a6793c0267ade5c955e70143"
server_key_version: "encryptionkey@0"
I20260924 13:46:12.677424 6395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "d295dd19b0604e7a8c182ee8b768cf13"
format_stamp: "Formatted at 2026-09-24 13:46:12 on dist-test-slave-tfm9"
server_key: "340e3f01ed0d409eba881ad957b9c313"
server_key_iv: "1be92201a6793c0267ade5c955e70143"
server_key_version: "encryptionkey@0"
I20260924 13:46:12.678781 6395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:46:12.679492 6410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:12.679654 6395 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:12.679754 6395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "d295dd19b0604e7a8c182ee8b768cf13"
format_stamp: "Formatted at 2026-09-24 13:46:12 on dist-test-slave-tfm9"
server_key: "340e3f01ed0d409eba881ad957b9c313"
server_key_iv: "1be92201a6793c0267ade5c955e70143"
server_key_version: "encryptionkey@0"
I20260924 13:46:12.679843 6395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:12.700832 6395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:12.703181 6395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:12.703351 6395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:12.707630 6395 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:38837
I20260924 13:46:12.707664 6462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:38837 every 8 connection(s)
I20260924 13:46:12.708011 6395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:12.708817 6463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:12.711169 6463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13: Bootstrap starting.
I20260924 13:46:12.711747 6463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:12.712052 6463 log.cc:824] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:12.712857 6463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13: No bootstrap required, opened a new log
I20260924 13:46:12.714515 6463 raft_consensus.cc:348] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d295dd19b0604e7a8c182ee8b768cf13" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38837 } }
I20260924 13:46:12.714653 6463 raft_consensus.cc:374] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:12.714675 6463 raft_consensus.cc:729] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d295dd19b0604e7a8c182ee8b768cf13, State: Initialized, Role: FOLLOWER
I20260924 13:46:12.714850 6463 consensus_queue.cc:260] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d295dd19b0604e7a8c182ee8b768cf13" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38837 } }
I20260924 13:46:12.714951 6463 raft_consensus.cc:388] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:12.714993 6463 raft_consensus.cc:482] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:12.715030 6463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:12.715540 6463 raft_consensus.cc:504] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d295dd19b0604e7a8c182ee8b768cf13" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38837 } }
I20260924 13:46:12.715654 6463 leader_election.cc:302] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d295dd19b0604e7a8c182ee8b768cf13; no voters:
I20260924 13:46:12.715845 6463 leader_election.cc:288] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:12.715900 6466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:12.716056 6466 raft_consensus.cc:686] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [term 1 LEADER]: Becoming Leader. State: Replica: d295dd19b0604e7a8c182ee8b768cf13, State: Running, Role: LEADER
I20260924 13:46:12.716113 6463 sys_catalog.cc:564] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:12.716204 6466 consensus_queue.cc:237] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d295dd19b0604e7a8c182ee8b768cf13" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38837 } }
I20260924 13:46:12.716773 6468 sys_catalog.cc:455] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d295dd19b0604e7a8c182ee8b768cf13. Latest consensus state: current_term: 1 leader_uuid: "d295dd19b0604e7a8c182ee8b768cf13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d295dd19b0604e7a8c182ee8b768cf13" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38837 } } }
I20260924 13:46:12.716890 6468 sys_catalog.cc:458] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:12.717060 6467 sys_catalog.cc:455] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d295dd19b0604e7a8c182ee8b768cf13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d295dd19b0604e7a8c182ee8b768cf13" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 38837 } } }
I20260924 13:46:12.717159 6467 sys_catalog.cc:458] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:12.717165 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 6395
I20260924 13:46:12.717299 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:12.717424 6475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:12.717517 19671 external_mini_cluster.cc:1428] Setting key 1e24152bc7276ab490a230f37d93e939
I20260924 13:46:12.717986 6475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:12.719591 6475 catalog_manager.cc:1347] Generated new cluster ID: 69ec31a9b70b4773976a735ba7d0f4c9
I20260924 13:46:12.719642 6475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:46:12 dist-test-slave-tfm9 krb5kdc[6382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:12.726820 6475 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:12.727288 6475 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:12.729180 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:12.717999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.19.53.254:38837)
0924 13:46:12.718180 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:12.718190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:12.718268 (+ 78us) server_negotiation.cc:407] Connection header received
0924 13:46:12.718592 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:12.718617 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:12.718687 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:12.718827 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:12.719597 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:12.720959 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:12.722062 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:12.722660 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:12.722755 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:12.725514 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:12.725519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:12.725519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:12.725531 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:12.727600 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:12.727995 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:12.727998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:12.727998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:12.728107 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:12.728309 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:12.728310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:12.728311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:12.728601 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:12.728757 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:12.728824 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 13:46:12.728966 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":59,"threads_started":1}
I20260924 13:46:12.732605 6475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13: Generated new TSK 0
I20260924 13:46:12.732760 6475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
Entry for principal kudu/127.19.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab.
I20260924 13:46:12.760596 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38837
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:12.841117 6489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:12.841360 6489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:12.841400 6489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:12.842610 6489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:12.842672 6489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:12.842715 6489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:46:12.844141 6489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.193.keytab
--principal=kudu/127.19.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:12.844409 6489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:12.844645 6489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:12.849521 6494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:12.849521 6497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:12.849558 6495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:12.849896 6489 server_base.cc:1034] running on GCE node
I20260924 13:46:12.850099 6489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:12.850289 6489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:12.851457 6489 hybrid_clock.cc:648] HybridClock initialized: now 1790257572851429 us; error 56 us; skew 500 ppm
Sep 24 13:46:12 dist-test-slave-tfm9 krb5kdc[6382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257572, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:12.853786 6489 init.cc:377] Logged in from keytab as kudu/127.19.53.193@KRBTEST.COM (short username kudu)
I20260924 13:46:12.854256 6489 webserver.cc:460] Webserver started at http://127.19.53.193:35731/ using document root <none> and password file <none>
I20260924 13:46:12.854480 6489 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:12.854549 6489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:12.854699 6489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:12.855787 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "5bf66fd1611848d982ebe50940eb4323"
format_stamp: "Formatted at 2026-09-24 13:46:12 on dist-test-slave-tfm9"
server_key: "27a2a51e025c74efd6da9964ba8b71bb"
server_key_iv: "80f32330bb671ec803a7ed215c5a94c7"
server_key_version: "encryptionkey@0"
I20260924 13:46:12.856151 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "5bf66fd1611848d982ebe50940eb4323"
format_stamp: "Formatted at 2026-09-24 13:46:12 on dist-test-slave-tfm9"
server_key: "27a2a51e025c74efd6da9964ba8b71bb"
server_key_iv: "80f32330bb671ec803a7ed215c5a94c7"
server_key_version: "encryptionkey@0"
I20260924 13:46:12.857438 6489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:12.858237 6504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:12.858414 6489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:46:12.858513 6489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "5bf66fd1611848d982ebe50940eb4323"
format_stamp: "Formatted at 2026-09-24 13:46:12 on dist-test-slave-tfm9"
server_key: "27a2a51e025c74efd6da9964ba8b71bb"
server_key_iv: "80f32330bb671ec803a7ed215c5a94c7"
server_key_version: "encryptionkey@0"
I20260924 13:46:12.858594 6489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:12.867122 6489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:12.868907 6489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:12.869061 6489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:12.869271 6489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:12.869580 6489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:12.869633 6489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:12.869666 6489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:12.869693 6489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:12.875414 6489 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:43247
I20260924 13:46:12.875442 6617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:43247 every 8 connection(s)
I20260924 13:46:12.875761 6489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:46:12 dist-test-slave-tfm9 krb5kdc[6382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257572, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.193@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:12.885052 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:12.876485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:47011 (local address 127.19.53.254:38837)
0924 13:46:12.876577 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:12.876579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:12.876768 (+ 189us) server_negotiation.cc:407] Connection header received
0924 13:46:12.877307 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:12.877308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:12.877321 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:12.877372 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:12.878390 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:12.878910 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:12.879907 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:12.880133 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:12.880212 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:12.882311 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:12.882312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:12.882313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:12.882320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:12.883649 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:12.884057 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:12.884058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:12.884058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:12.884124 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:12.884392 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:12.884393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:12.884394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:12.884517 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:12.884596 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:12.884872 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 13:46:12.884962 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 13:46:12.885226 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:12.876454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38837 (local address 127.19.53.193:47011)
0924 13:46:12.876625 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:12.876692 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:12.877187 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:12.877452 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:12.877456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:12.878082 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:12.878284 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:12.878289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:12.879011 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:12.879013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:12.879811 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:12.879813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:12.880220 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:12.880221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:12.880354 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:12.880863 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:12.880870 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:12.882200 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:12.883739 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:12.883746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:12.883748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:12.883971 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:12.884190 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:12.884191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:12.884191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:12.884306 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:12.884586 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:12.884593 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:12.884732 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 13:46:12.884841 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 13:46:12.884996 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260924 13:46:12.885638 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 6489
I20260924 13:46:12.885763 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:46:12.885856 6618 heartbeater.cc:347] Connected to a master server at 127.19.53.254:38837
I20260924 13:46:12.885977 6618 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:12.886024 19671 external_mini_cluster.cc:1428] Setting key 0d888f3428765ec5fcf0b34e90a15b91
I20260924 13:46:12.886160 6618 heartbeater.cc:511] Master 127.19.53.254:38837 requested a full tablet report, sending...
I20260924 13:46:12.886803 6426 ts_manager.cc:194] Registered new tserver with Master: 5bf66fd1611848d982ebe50940eb4323 (127.19.53.193:43247)
I20260924 13:46:12.887720 6426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.193@KRBTEST.COM'} at 127.19.53.193:47011
WARNING: no policy specified for kudu/127.19.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
Entry for principal kudu/127.19.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab.
I20260924 13:46:12.910377 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38837
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:12.989552 6623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:12.989741 6623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:12.989763 6623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:12.990919 6623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:12.990959 6623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:12.991014 6623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:46:12.992430 6623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.194.keytab
--principal=kudu/127.19.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:12.992681 6623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:12.992908 6623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:12.997818 6631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:12.997825 6628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:12.998082 6623 server_base.cc:1034] running on GCE node
W20260924 13:46:12.997872 6629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:12.998365 6623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:12.998618 6623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:12.999754 6623 hybrid_clock.cc:648] HybridClock initialized: now 1790257572999738 us; error 42 us; skew 500 ppm
Sep 24 13:46:12 dist-test-slave-tfm9 krb5kdc[6382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257572, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:13.002110 6623 init.cc:377] Logged in from keytab as kudu/127.19.53.194@KRBTEST.COM (short username kudu)
I20260924 13:46:13.002550 6623 webserver.cc:460] Webserver started at http://127.19.53.194:46851/ using document root <none> and password file <none>
I20260924 13:46:13.002780 6623 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:13.002856 6623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:13.002983 6623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:13.003990 6623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "d6f7e34620e4426c9fc3484f1e1bd92a"
format_stamp: "Formatted at 2026-09-24 13:46:12 on dist-test-slave-tfm9"
server_key: "9b2b629d12454e1f9d482b5a701f59e6"
server_key_iv: "fda8af64bb148504e326a2988ec623c3"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.004348 6623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "d6f7e34620e4426c9fc3484f1e1bd92a"
format_stamp: "Formatted at 2026-09-24 13:46:12 on dist-test-slave-tfm9"
server_key: "9b2b629d12454e1f9d482b5a701f59e6"
server_key_iv: "fda8af64bb148504e326a2988ec623c3"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.005642 6623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:46:13.006444 6638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.006605 6623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.006711 6623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "d6f7e34620e4426c9fc3484f1e1bd92a"
format_stamp: "Formatted at 2026-09-24 13:46:12 on dist-test-slave-tfm9"
server_key: "9b2b629d12454e1f9d482b5a701f59e6"
server_key_iv: "fda8af64bb148504e326a2988ec623c3"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.006798 6623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:13.027585 6623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:13.029531 6623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:13.029690 6623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:13.030035 6623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:13.030355 6623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:13.030409 6623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.030450 6623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:13.030476 6623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.036454 6623 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:43773
I20260924 13:46:13.036513 6751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:43773 every 8 connection(s)
I20260924 13:46:13.036832 6623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:46:13 dist-test-slave-tfm9 krb5kdc[6382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257572, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.194@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:13.045020 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 6623
I20260924 13:46:13.045105 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:46:13.045287 19671 external_mini_cluster.cc:1428] Setting key b10148b7386f6435b76201705a3573cc
I20260924 13:46:13.046947 6754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.037570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38837 (local address 127.19.53.194:55273)
0924 13:46:13.037758 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:13.037794 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:13.038367 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:13.038643 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:13.038652 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:13.039234 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:13.039429 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.039434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.040242 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.040244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.041080 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.041082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.041509 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.041510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.041645 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.042160 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:13.042168 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:13.043788 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:13.045435 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:13.045442 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:13.045444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:13.045743 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:13.046013 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:13.046014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:13.046015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:13.046170 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:13.046463 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:13.046467 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:13.046584 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 13:46:13.046666 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 13:46:13.046783 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260924 13:46:13.046945 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.037567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:55273 (local address 127.19.53.254:38837)
0924 13:46:13.037645 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:13.037647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:13.037886 (+ 239us) server_negotiation.cc:407] Connection header received
0924 13:46:13.038485 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:13.038486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:13.038501 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:13.038571 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:13.039559 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.040127 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.041183 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.041401 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.041496 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.043887 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:13.043888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:13.043889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:13.043896 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:13.045350 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:13.045845 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:13.045846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:13.045847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:13.045947 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:13.046241 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:13.046242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:13.046243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:13.046396 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:13.046467 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:13.046710 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 13:46:13.046804 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 13:46:13.047518 6752 heartbeater.cc:347] Connected to a master server at 127.19.53.254:38837
I20260924 13:46:13.047629 6752 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:13.047824 6752 heartbeater.cc:511] Master 127.19.53.254:38837 requested a full tablet report, sending...
I20260924 13:46:13.048214 6426 ts_manager.cc:194] Registered new tserver with Master: d6f7e34620e4426c9fc3484f1e1bd92a (127.19.53.194:43773)
I20260924 13:46:13.048619 6426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.194@KRBTEST.COM'} at 127.19.53.194:55273
WARNING: no policy specified for kudu/127.19.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
Entry for principal kudu/127.19.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab.
I20260924 13:46:13.071946 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38837
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:13.154685 6757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:13.154879 6757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:13.154902 6757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:13.155952 6757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:13.155985 6757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:13.156010 6757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:46:13.157322 6757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.195.keytab
--principal=kudu/127.19.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:13.157483 6757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:13.157634 6757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:13.162412 6765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:13.162405 6763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:13.162403 6762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:13.162582 6757 server_base.cc:1034] running on GCE node
I20260924 13:46:13.162808 6757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:13.163043 6757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:13.164192 6757 hybrid_clock.cc:648] HybridClock initialized: now 1790257573164186 us; error 36 us; skew 500 ppm
Sep 24 13:46:13 dist-test-slave-tfm9 krb5kdc[6382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257573, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:13.166548 6757 init.cc:377] Logged in from keytab as kudu/127.19.53.195@KRBTEST.COM (short username kudu)
I20260924 13:46:13.167039 6757 webserver.cc:460] Webserver started at http://127.19.53.195:39231/ using document root <none> and password file <none>
I20260924 13:46:13.167261 6757 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:13.167335 6757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:13.167486 6757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:13.168499 6757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "6500ce2c2bb64dcca70133f56f9e0c4c"
format_stamp: "Formatted at 2026-09-24 13:46:13 on dist-test-slave-tfm9"
server_key: "f56bd2a117d34ccfce02909912a3a0c7"
server_key_iv: "b7e21c025721602fde3bc63a0ba6d6cf"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.168870 6757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "6500ce2c2bb64dcca70133f56f9e0c4c"
format_stamp: "Formatted at 2026-09-24 13:46:13 on dist-test-slave-tfm9"
server_key: "f56bd2a117d34ccfce02909912a3a0c7"
server_key_iv: "b7e21c025721602fde3bc63a0ba6d6cf"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.170315 6757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:46:13.171062 6772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.171219 6757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:13.171320 6757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "6500ce2c2bb64dcca70133f56f9e0c4c"
format_stamp: "Formatted at 2026-09-24 13:46:13 on dist-test-slave-tfm9"
server_key: "f56bd2a117d34ccfce02909912a3a0c7"
server_key_iv: "b7e21c025721602fde3bc63a0ba6d6cf"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.171391 6757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:13.177529 6757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:13.179293 6757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:13.179474 6757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:13.179687 6757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:13.179986 6757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:13.180039 6757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.180073 6757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:13.180100 6757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.186184 6757 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:39509
I20260924 13:46:13.186231 6885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:39509 every 8 connection(s)
I20260924 13:46:13.186546 6757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:46:13 dist-test-slave-tfm9 krb5kdc[6382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257573, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.195@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:13.195892 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.187237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:51575 (local address 127.19.53.254:38837)
0924 13:46:13.187283 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:13.187285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:13.187528 (+ 243us) server_negotiation.cc:407] Connection header received
0924 13:46:13.188009 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:13.188010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:13.188021 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:13.188075 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:13.188976 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.189498 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.190674 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.190874 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.190940 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.193177 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:13.193179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:13.193179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:13.193186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:13.194680 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:13.195056 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:13.195057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:13.195057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:13.195115 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:13.195334 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:13.195334 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:13.195335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:13.195450 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:13.195509 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:13.195754 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 13:46:13.195825 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:46:13.196009 6888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.187281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:38837 (local address 127.19.53.195:51575)
0924 13:46:13.187427 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:13.187458 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:13.187900 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:13.188126 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:13.188129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:13.188694 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:13.188897 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.188902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.189598 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.189599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.190581 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.190584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.190955 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.190956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.191055 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.191525 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:13.191534 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:13.193036 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:13.194759 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:13.194765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:13.194767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:13.194983 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:13.195178 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:13.195179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:13.195179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:13.195273 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:13.195509 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:13.195513 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:13.195621 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 13:46:13.195702 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 13:46:13.195819 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":90,"threads_started":1}
I20260924 13:46:13.196506 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 6757
I20260924 13:46:13.196630 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:46:13.196660 6886 heartbeater.cc:347] Connected to a master server at 127.19.53.254:38837
I20260924 13:46:13.196748 6886 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:13.196825 19671 external_mini_cluster.cc:1428] Setting key df41f88b3df966e5e428bab338898aed
I20260924 13:46:13.196939 6886 heartbeater.cc:511] Master 127.19.53.254:38837 requested a full tablet report, sending...
I20260924 13:46:13.197330 6426 ts_manager.cc:194] Registered new tserver with Master: 6500ce2c2bb64dcca70133f56f9e0c4c (127.19.53.195:39509)
I20260924 13:46:13.197732 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:46:13.197796 6426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.53.195@KRBTEST.COM'} at 127.19.53.195:51575
I20260924 13:46:13.205250 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.199155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47388 (local address 127.19.53.254:38837)
0924 13:46:13.199241 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:13.199243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:13.199363 (+ 120us) server_negotiation.cc:407] Connection header received
0924 13:46:13.199424 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:13.199425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:13.199435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:13.199491 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:13.200042 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.200517 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.201224 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.201413 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.201483 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.202654 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:13.202655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:13.202655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:13.202661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:13.204025 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:13.204452 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:13.204453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:13.204453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:13.204511 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:13.204709 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:13.204710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:13.204710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:13.204854 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:13.204957 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:13.205081 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 13:46:13.205153 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 13:46:13.206859 6426 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:46:13.208338 6426 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:46:13.214514 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.210404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:43247 (local address 127.0.0.1:56780)
0924 13:46:13.210663 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:13.210672 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:13.210722 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:13.211063 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:13.211064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:13.211066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:13.211121 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.211123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.212083 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.212083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.213335 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.213338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.214183 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.214184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.214270 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.214284 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 13:46:13.214388 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 13:46:13.214443 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":67,"threads_started":1}
I20260924 13:46:13.214527 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.210499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.19.53.193:43247)
0924 13:46:13.210897 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:13.210902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:13.210910 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:46:13.210946 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:13.210949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:13.210965 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:13.211197 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:13.211271 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.211978 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.213422 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.214088 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.214180 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.214242 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:13.214407 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 13:46:13.214451 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":78,"threads_started":1}
I20260924 13:46:13.214704 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.210379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:43773 (local address 127.0.0.1:33222)
0924 13:46:13.210517 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:13.210553 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:13.210619 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:13.211228 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:13.211229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:13.211231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:13.211264 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.211265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.212543 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.212544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.213576 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.213579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.214452 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.214453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.214524 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.214528 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:13.214560 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 13:46:13.214607 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260924 13:46:13.215085 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.210729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.19.53.194:43773)
0924 13:46:13.210940 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:13.210944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:13.210951 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:46:13.211104 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:13.211110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:13.211130 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:13.211330 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:13.211401 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.212432 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.213648 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.214358 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.214856 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.214935 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:13.214961 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 13:46:13.215006 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
I20260924 13:46:13.215269 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.210582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:39509 (local address 127.0.0.1:34438)
0924 13:46:13.210806 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:13.210814 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:13.210848 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:13.211036 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:13.211039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:13.211041 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:13.211107 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.211109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.212416 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.212417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.213482 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.213489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.214483 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.214484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.214533 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.214536 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:46:13.215021 (+ 485us) client_negotiation.cc:240] Negotiation successful
0924 13:46:13.215067 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":58,"threads_started":1}
I20260924 13:46:13.215272 6552 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff1ebe6e832478b9655f5e5032acd55 (DEFAULT_TABLE table=test-table [id=5f2385a050144c79a7bb6ca3bbb9e9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:13.215580 6898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.210675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.19.53.195:39509)
0924 13:46:13.210856 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:13.210861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:13.210873 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:46:13.210909 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:13.210911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:13.210928 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:13.211015 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:13.211194 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.212309 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.213565 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.214320 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.214396 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.214455 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:13.215450 (+ 995us) server_negotiation.cc:299] Negotiation successful
0924 13:46:13.215499 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260924 13:46:13.215658 6552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff1ebe6e832478b9655f5e5032acd55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:13.215796 6686 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff1ebe6e832478b9655f5e5032acd55 (DEFAULT_TABLE table=test-table [id=5f2385a050144c79a7bb6ca3bbb9e9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:13.216135 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff1ebe6e832478b9655f5e5032acd55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:13.219250 6901 tablet_bootstrap.cc:492] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a: Bootstrap starting.
I20260924 13:46:13.219250 6900 tablet_bootstrap.cc:492] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323: Bootstrap starting.
I20260924 13:46:13.219640 6820 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff1ebe6e832478b9655f5e5032acd55 (DEFAULT_TABLE table=test-table [id=5f2385a050144c79a7bb6ca3bbb9e9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:13.219941 6820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff1ebe6e832478b9655f5e5032acd55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:13.219967 6901 tablet_bootstrap.cc:654] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:13.219967 6900 tablet_bootstrap.cc:654] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:13.220614 6901 log.cc:824] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:13.220724 6900 log.cc:824] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:13.221378 6901 tablet_bootstrap.cc:492] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a: No bootstrap required, opened a new log
I20260924 13:46:13.221462 6901 ts_tablet_manager.cc:1397] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:46:13.221535 6900 tablet_bootstrap.cc:492] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323: No bootstrap required, opened a new log
I20260924 13:46:13.221609 6900 ts_tablet_manager.cc:1397] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:46:13.222944 6904 tablet_bootstrap.cc:492] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c: Bootstrap starting.
I20260924 13:46:13.222929 6901 raft_consensus.cc:348] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f7e34620e4426c9fc3484f1e1bd92a" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43773 } } peers { permanent_uuid: "5bf66fd1611848d982ebe50940eb4323" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43247 } } peers { permanent_uuid: "6500ce2c2bb64dcca70133f56f9e0c4c" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39509 } }
I20260924 13:46:13.223109 6901 raft_consensus.cc:374] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:13.223148 6901 raft_consensus.cc:729] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6f7e34620e4426c9fc3484f1e1bd92a, State: Initialized, Role: FOLLOWER
I20260924 13:46:13.223256 6901 consensus_queue.cc:260] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f7e34620e4426c9fc3484f1e1bd92a" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43773 } } peers { permanent_uuid: "5bf66fd1611848d982ebe50940eb4323" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43247 } } peers { permanent_uuid: "6500ce2c2bb64dcca70133f56f9e0c4c" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39509 } }
I20260924 13:46:13.223562 6901 ts_tablet_manager.cc:1428] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:46:13.223729 6752 heartbeater.cc:503] Master 127.19.53.254:38837 was elected leader, sending a full tablet report...
I20260924 13:46:13.223701 6900 raft_consensus.cc:348] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f7e34620e4426c9fc3484f1e1bd92a" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43773 } } peers { permanent_uuid: "5bf66fd1611848d982ebe50940eb4323" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43247 } } peers { permanent_uuid: "6500ce2c2bb64dcca70133f56f9e0c4c" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39509 } }
I20260924 13:46:13.223881 6900 raft_consensus.cc:374] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:13.223922 6900 raft_consensus.cc:729] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf66fd1611848d982ebe50940eb4323, State: Initialized, Role: FOLLOWER
I20260924 13:46:13.224005 6904 tablet_bootstrap.cc:654] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:13.224036 6900 consensus_queue.cc:260] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f7e34620e4426c9fc3484f1e1bd92a" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43773 } } peers { permanent_uuid: "5bf66fd1611848d982ebe50940eb4323" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43247 } } peers { permanent_uuid: "6500ce2c2bb64dcca70133f56f9e0c4c" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39509 } }
I20260924 13:46:13.224330 6618 heartbeater.cc:503] Master 127.19.53.254:38837 was elected leader, sending a full tablet report...
I20260924 13:46:13.224303 6900 ts_tablet_manager.cc:1428] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:46:13.224538 6904 log.cc:824] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:13.225256 6904 tablet_bootstrap.cc:492] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c: No bootstrap required, opened a new log
I20260924 13:46:13.225342 6904 ts_tablet_manager.cc:1397] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:46:13.226805 6904 raft_consensus.cc:348] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f7e34620e4426c9fc3484f1e1bd92a" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43773 } } peers { permanent_uuid: "5bf66fd1611848d982ebe50940eb4323" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43247 } } peers { permanent_uuid: "6500ce2c2bb64dcca70133f56f9e0c4c" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39509 } }
I20260924 13:46:13.226934 6904 raft_consensus.cc:374] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:13.226956 6904 raft_consensus.cc:729] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6500ce2c2bb64dcca70133f56f9e0c4c, State: Initialized, Role: FOLLOWER
I20260924 13:46:13.227089 6904 consensus_queue.cc:260] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f7e34620e4426c9fc3484f1e1bd92a" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43773 } } peers { permanent_uuid: "5bf66fd1611848d982ebe50940eb4323" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43247 } } peers { permanent_uuid: "6500ce2c2bb64dcca70133f56f9e0c4c" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39509 } }
I20260924 13:46:13.227361 6886 heartbeater.cc:503] Master 127.19.53.254:38837 was elected leader, sending a full tablet report...
I20260924 13:46:13.227346 6904 ts_tablet_manager.cc:1428] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260924 13:46:13.287420 6753 tablet.cc:2369] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:46:13.376581 6619 tablet.cc:2369] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:46:13.437160 6887 tablet.cc:2369] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:46:13.477536 6906 raft_consensus.cc:482] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:46:13.477653 6906 raft_consensus.cc:504] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f7e34620e4426c9fc3484f1e1bd92a" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43773 } } peers { permanent_uuid: "5bf66fd1611848d982ebe50940eb4323" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43247 } } peers { permanent_uuid: "6500ce2c2bb64dcca70133f56f9e0c4c" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39509 } }
I20260924 13:46:13.478138 6906 leader_election.cc:288] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6f7e34620e4426c9fc3484f1e1bd92a (127.19.53.194:43773), 6500ce2c2bb64dcca70133f56f9e0c4c (127.19.53.195:39509)
I20260924 13:46:13.481004 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.478370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:52533 (local address 127.19.53.194:43773)
0924 13:46:13.478416 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:13.478419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:13.478497 (+ 78us) server_negotiation.cc:407] Connection header received
0924 13:46:13.478571 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:13.478571 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:13.478583 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:13.478660 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:13.478846 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.479314 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.480267 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.480645 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.480734 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.480743 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:13.480900 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 13:46:13.480936 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:46:13.481029 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.478307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:43773 (local address 127.19.53.193:52533)
0924 13:46:13.478442 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:13.478451 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:13.478514 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:13.478664 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:13.478664 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:13.478666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:13.478742 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.478744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.479410 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.479410 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.480171 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.480173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.480732 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.480733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.480803 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.480807 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:13.480892 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 13:46:13.480933 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20260924 13:46:13.481031 6898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.478376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:55165 (local address 127.19.53.195:39509)
0924 13:46:13.478422 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:13.478424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:13.478564 (+ 140us) server_negotiation.cc:407] Connection header received
0924 13:46:13.478630 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:13.478631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:13.478640 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:13.478708 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:13.478902 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.479363 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.480294 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.480678 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.480763 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.480771 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:13.480919 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 13:46:13.480960 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:46:13.481051 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.478308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:39509 (local address 127.19.53.193:55165)
0924 13:46:13.478483 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:13.478491 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:13.478570 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:13.478725 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:13.478725 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:13.478727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:13.478825 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.478826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.479448 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.479449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.480198 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.480200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.480762 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.480763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.480842 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.480846 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:13.480912 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 13:46:13.480965 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":14,"thread_start_us":74,"threads_started":1}
I20260924 13:46:13.481472 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff1ebe6e832478b9655f5e5032acd55" candidate_uuid: "5bf66fd1611848d982ebe50940eb4323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6500ce2c2bb64dcca70133f56f9e0c4c" is_pre_election: true
I20260924 13:46:13.481617 6840 raft_consensus.cc:2454] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf66fd1611848d982ebe50940eb4323 in term 0.
I20260924 13:46:13.481570 6706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff1ebe6e832478b9655f5e5032acd55" candidate_uuid: "5bf66fd1611848d982ebe50940eb4323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f7e34620e4426c9fc3484f1e1bd92a" is_pre_election: true
I20260924 13:46:13.481727 6706 raft_consensus.cc:2454] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf66fd1611848d982ebe50940eb4323 in term 0.
I20260924 13:46:13.481837 6507 leader_election.cc:302] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf66fd1611848d982ebe50940eb4323, 6500ce2c2bb64dcca70133f56f9e0c4c; no voters:
I20260924 13:46:13.481980 6906 raft_consensus.cc:2781] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:46:13.482038 6906 raft_consensus.cc:482] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:46:13.482060 6906 raft_consensus.cc:3037] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:13.482733 6906 raft_consensus.cc:504] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f7e34620e4426c9fc3484f1e1bd92a" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43773 } } peers { permanent_uuid: "5bf66fd1611848d982ebe50940eb4323" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43247 } } peers { permanent_uuid: "6500ce2c2bb64dcca70133f56f9e0c4c" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39509 } }
I20260924 13:46:13.482892 6906 leader_election.cc:288] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [CANDIDATE]: Term 1 election: Requested vote from peers d6f7e34620e4426c9fc3484f1e1bd92a (127.19.53.194:43773), 6500ce2c2bb64dcca70133f56f9e0c4c (127.19.53.195:39509)
I20260924 13:46:13.482969 6706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff1ebe6e832478b9655f5e5032acd55" candidate_uuid: "5bf66fd1611848d982ebe50940eb4323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f7e34620e4426c9fc3484f1e1bd92a"
I20260924 13:46:13.483071 6706 raft_consensus.cc:3037] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:13.483094 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff1ebe6e832478b9655f5e5032acd55" candidate_uuid: "5bf66fd1611848d982ebe50940eb4323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6500ce2c2bb64dcca70133f56f9e0c4c"
I20260924 13:46:13.483177 6840 raft_consensus.cc:3037] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:13.483873 6706 raft_consensus.cc:2454] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf66fd1611848d982ebe50940eb4323 in term 1.
I20260924 13:46:13.483928 6840 raft_consensus.cc:2454] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf66fd1611848d982ebe50940eb4323 in term 1.
I20260924 13:46:13.484050 6506 leader_election.cc:302] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf66fd1611848d982ebe50940eb4323, d6f7e34620e4426c9fc3484f1e1bd92a; no voters:
I20260924 13:46:13.484176 6906 raft_consensus.cc:2781] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:13.484352 6906 raft_consensus.cc:686] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [term 1 LEADER]: Becoming Leader. State: Replica: 5bf66fd1611848d982ebe50940eb4323, State: Running, Role: LEADER
I20260924 13:46:13.484491 6906 consensus_queue.cc:237] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f7e34620e4426c9fc3484f1e1bd92a" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43773 } } peers { permanent_uuid: "5bf66fd1611848d982ebe50940eb4323" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43247 } } peers { permanent_uuid: "6500ce2c2bb64dcca70133f56f9e0c4c" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39509 } }
I20260924 13:46:13.485351 6426 catalog_manager.cc:5495] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bf66fd1611848d982ebe50940eb4323 (127.19.53.193). New cstate: current_term: 1 leader_uuid: "5bf66fd1611848d982ebe50940eb4323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f7e34620e4426c9fc3484f1e1bd92a" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bf66fd1611848d982ebe50940eb4323" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 43247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6500ce2c2bb64dcca70133f56f9e0c4c" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39509 } health_report { overall_health: UNKNOWN } } }
I20260924 13:46:13.503769 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.501512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.19.53.193:43247)
0924 13:46:13.501571 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:13.501572 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:13.501583 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:46:13.501649 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:13.501650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:13.501668 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:13.501729 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:13.501870 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.502384 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.503102 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.503282 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.503339 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.503472 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:13.503566 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:13.503622 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:13.503658 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 13:46:13.503673 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:46:13.505751 6840 raft_consensus.cc:1261] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c [term 1 FOLLOWER]: Refusing update from remote peer 5bf66fd1611848d982ebe50940eb4323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:13.505751 6706 raft_consensus.cc:1261] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a [term 1 FOLLOWER]: Refusing update from remote peer 5bf66fd1611848d982ebe50940eb4323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:13.506150 6906 consensus_queue.cc:1035] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6f7e34620e4426c9fc3484f1e1bd92a" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 43773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:13.506310 6911 consensus_queue.cc:1035] T 4ff1ebe6e832478b9655f5e5032acd55 P 5bf66fd1611848d982ebe50940eb4323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6500ce2c2bb64dcca70133f56f9e0c4c" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 39509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:13.508637 6914 mvcc.cc:205] Tried to move back new op lower bound from 7332895021076398080 to 7332895020993146880. Current Snapshot: MvccSnapshot[applied={T|T < 7332895021076398080}]
I20260924 13:46:13.509663 6916 mvcc.cc:205] Tried to move back new op lower bound from 7332895021076398080 to 7332895020993146880. Current Snapshot: MvccSnapshot[applied={T|T < 7332895021076398080}]
I20260924 13:46:13.509600 6915 mvcc.cc:205] Tried to move back new op lower bound from 7332895021076398080 to 7332895020993146880. Current Snapshot: MvccSnapshot[applied={T|T < 7332895021076398080 or (T in {7332895021076398080})}]
I20260924 13:46:13.511149 6426 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:46:13.511246 6426 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:46:13.512044 6426 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d295dd19b0604e7a8c182ee8b768cf13: Sending DeleteTablet for 3 replicas of tablet 4ff1ebe6e832478b9655f5e5032acd55
I20260924 13:46:13.512373 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ff1ebe6e832478b9655f5e5032acd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:13 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:33222
I20260924 13:46:13.512400 6820 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ff1ebe6e832478b9655f5e5032acd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:13 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:34438
I20260924 13:46:13.512593 6924 tablet_replica.cc:330] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a: stopping tablet replica
I20260924 13:46:13.512639 6552 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ff1ebe6e832478b9655f5e5032acd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:13 UTC) from {username='kudu', principal='kudu/127.19.53.254@KRBTEST.COM'} at 127.0.0.1:56780
I20260924 13:46:13.512687 6924 raft_consensus.cc:2229] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:46:13.512795 6924 raft_consensus.cc:2258] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:46:13.512898 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 6489
I20260924 13:46:13.513805 6924 ts_tablet_manager.cc:1905] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:46:13.514555 6925 tablet_replica.cc:330] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c: stopping tablet replica
I20260924 13:46:13.514671 6925 raft_consensus.cc:2229] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:46:13.514771 6925 raft_consensus.cc:2258] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:46:13.515305 6925 ts_tablet_manager.cc:1905] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:46:13.515882 6924 ts_tablet_manager.cc:1918] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:46:13.515957 6924 log.cc:1196] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/4ff1ebe6e832478b9655f5e5032acd55
I20260924 13:46:13.516223 6924 ts_tablet_manager.cc:1939] T 4ff1ebe6e832478b9655f5e5032acd55 P d6f7e34620e4426c9fc3484f1e1bd92a: Deleting consensus metadata
I20260924 13:46:13.516636 6412 catalog_manager.cc:4841] TS d6f7e34620e4426c9fc3484f1e1bd92a (127.19.53.194:43773): tablet 4ff1ebe6e832478b9655f5e5032acd55 (table test-table [id=5f2385a050144c79a7bb6ca3bbb9e9f7]) successfully deleted
I20260924 13:46:13.517027 6925 ts_tablet_manager.cc:1918] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:46:13.517083 6925 log.cc:1196] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/4ff1ebe6e832478b9655f5e5032acd55
I20260924 13:46:13.517282 6925 ts_tablet_manager.cc:1939] T 4ff1ebe6e832478b9655f5e5032acd55 P 6500ce2c2bb64dcca70133f56f9e0c4c: Deleting consensus metadata
I20260924 13:46:13.517634 6413 catalog_manager.cc:4841] TS 6500ce2c2bb64dcca70133f56f9e0c4c (127.19.53.195:39509): tablet 4ff1ebe6e832478b9655f5e5032acd55 (table test-table [id=5f2385a050144c79a7bb6ca3bbb9e9f7]) successfully deleted
W20260924 13:46:13.519544 6413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:43247 (error 108)
I20260924 13:46:13.520053 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.519916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:43247 (local address 127.0.0.1:56798)
0924 13:46:13.519975 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:13.520012 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 13:46:13.520144 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 6623
W20260924 13:46:13.520155 6413 catalog_manager.cc:4584] TS 5bf66fd1611848d982ebe50940eb4323 (127.19.53.193:43247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ff1ebe6e832478b9655f5e5032acd55: Network error: Client connection negotiation failed: client connection to 127.19.53.193:43247: connect: Connection refused (error 111)
I20260924 13:46:13.525610 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 6757
I20260924 13:46:13.530084 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 6395
2026-09-24T13:46:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2102 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T13:46:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:13Z Disabled control of system clock
I20260924 13:46:13.557438 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:41067
--webserver_interface=127.19.53.254
--webserver_port=0
--builtin_ntp_servers=127.19.53.212:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:41067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:46:13.636507 6931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:13.636711 6931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:13.636730 6931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:13.637851 6931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:13.637892 6931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:13.637904 6931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:13.637979 6931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:13.637995 6931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:13.639449 6931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:41067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:41067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:13.639727 6931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:13.640025 6931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:13.644790 6937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:13.644984 6931 server_base.cc:1034] running on GCE node
W20260924 13:46:13.644879 6936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:13.644790 6939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:13.645327 6931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:13.645617 6931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:13.646782 6931 hybrid_clock.cc:648] HybridClock initialized: now 1790257573646757 us; error 40 us; skew 500 ppm
I20260924 13:46:13.648063 6931 webserver.cc:460] Webserver started at http://127.19.53.254:39751/ using document root <none> and password file <none>
I20260924 13:46:13.648327 6931 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:13.648380 6931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:13.648564 6931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:13.649711 6931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "8160620f46f1486b894455006f6f5c82"
format_stamp: "Formatted at 2026-09-24 13:46:13 on dist-test-slave-tfm9"
server_key: "652d5a1d493c9bc79dfef7d0c9ccf6bf"
server_key_iv: "2316657a6371d1729e7964d5f75fcb24"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.650126 6931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "8160620f46f1486b894455006f6f5c82"
format_stamp: "Formatted at 2026-09-24 13:46:13 on dist-test-slave-tfm9"
server_key: "652d5a1d493c9bc79dfef7d0c9ccf6bf"
server_key_iv: "2316657a6371d1729e7964d5f75fcb24"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.651563 6931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:46:13.652348 6945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.652503 6931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:46:13.652585 6931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "8160620f46f1486b894455006f6f5c82"
format_stamp: "Formatted at 2026-09-24 13:46:13 on dist-test-slave-tfm9"
server_key: "652d5a1d493c9bc79dfef7d0c9ccf6bf"
server_key_iv: "2316657a6371d1729e7964d5f75fcb24"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.652639 6931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:13.673429 6931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:13.673683 6931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:13.673784 6931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:13.677306 6931 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:41067
I20260924 13:46:13.677392 6997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:41067 every 8 connection(s)
I20260924 13:46:13.677690 6931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:13.678665 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:13.680850 6998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82: Bootstrap starting.
I20260924 13:46:13.681433 6998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:13.681563 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 6931
I20260924 13:46:13.681689 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:13.681795 6998 log.cc:824] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:13.681948 19671 external_mini_cluster.cc:1428] Setting key 4f0770376316b1edb7d4ddfae3e6dc95
I20260924 13:46:13.682561 6998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82: No bootstrap required, opened a new log
I20260924 13:46:13.684230 6998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8160620f46f1486b894455006f6f5c82" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41067 } }
I20260924 13:46:13.684361 6998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:13.684408 6998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8160620f46f1486b894455006f6f5c82, State: Initialized, Role: FOLLOWER
I20260924 13:46:13.684581 6998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8160620f46f1486b894455006f6f5c82" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41067 } }
I20260924 13:46:13.684674 6998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:13.684708 6998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:13.684757 6998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:13.685591 6998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8160620f46f1486b894455006f6f5c82" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41067 } }
I20260924 13:46:13.685688 6998 leader_election.cc:302] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8160620f46f1486b894455006f6f5c82; no voters:
I20260924 13:46:13.685868 6998 leader_election.cc:288] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:13.685892 7003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:13.686020 7003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [term 1 LEADER]: Becoming Leader. State: Replica: 8160620f46f1486b894455006f6f5c82, State: Running, Role: LEADER
I20260924 13:46:13.686128 6998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:13.686096 7003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8160620f46f1486b894455006f6f5c82" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41067 } }
I20260924 13:46:13.686668 7005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8160620f46f1486b894455006f6f5c82. Latest consensus state: current_term: 1 leader_uuid: "8160620f46f1486b894455006f6f5c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8160620f46f1486b894455006f6f5c82" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41067 } } }
I20260924 13:46:13.686904 7005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:13.686833 7004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8160620f46f1486b894455006f6f5c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8160620f46f1486b894455006f6f5c82" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 41067 } } }
I20260924 13:46:13.687413 7013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:13.687906 7013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:13.687402 7004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:13.689191 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.682618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.19.53.254:41067)
0924 13:46:13.682804 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:13.682811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:13.682835 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:46:13.684218 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:13.684224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:13.684288 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:13.684458 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:13.684717 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.685763 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.686858 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.687511 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.687627 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.688329 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:13.688333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:13.688334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:13.688526 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:13.688567 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:13.688582 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:13.688645 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:13.688887 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 13:46:13.688971 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260924 13:46:13.689538 7013 catalog_manager.cc:1347] Generated new cluster ID: b361efe70fc6429aaf38fbede9d06311
I20260924 13:46:13.689587 7013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:13.729544 7013 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:13.730168 7013 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:13.732663 7013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82: Generated new TSK 0
I20260924 13:46:13.732846 7013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:13.737242 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.193:0
--local_ip_for_outbound_sockets=127.19.53.193
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41067
--builtin_ntp_servers=127.19.53.212:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:46:13.827683 7022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:13.827916 7022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:13.827953 7022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:13.829089 7022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:13.829160 7022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:13.829200 7022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.193
I20260924 13:46:13.830574 7022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.53.193
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.193
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:13.830834 7022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:13.831146 7022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:13.836148 7030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:13.836160 7028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:13.836184 7027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:13.836464 7022 server_base.cc:1034] running on GCE node
I20260924 13:46:13.836684 7022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:13.836918 7022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:13.838053 7022 hybrid_clock.cc:648] HybridClock initialized: now 1790257573838039 us; error 30 us; skew 500 ppm
I20260924 13:46:13.839386 7022 webserver.cc:460] Webserver started at http://127.19.53.193:42523/ using document root <none> and password file <none>
I20260924 13:46:13.839644 7022 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:13.839687 7022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:13.839851 7022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:13.840880 7022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/data/instance:
uuid: "0dd3f41bf9934cd99e29c2859bee90d9"
format_stamp: "Formatted at 2026-09-24 13:46:13 on dist-test-slave-tfm9"
server_key: "bf5b1610335b8b3574dee8ffc27d310f"
server_key_iv: "b84e01786241ac7b6d58e954cf2222ce"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.841226 7022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance:
uuid: "0dd3f41bf9934cd99e29c2859bee90d9"
format_stamp: "Formatted at 2026-09-24 13:46:13 on dist-test-slave-tfm9"
server_key: "bf5b1610335b8b3574dee8ffc27d310f"
server_key_iv: "b84e01786241ac7b6d58e954cf2222ce"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.842612 7022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:13.843299 7036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.843500 7022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:46:13.843612 7022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/wal
uuid: "0dd3f41bf9934cd99e29c2859bee90d9"
format_stamp: "Formatted at 2026-09-24 13:46:13 on dist-test-slave-tfm9"
server_key: "bf5b1610335b8b3574dee8ffc27d310f"
server_key_iv: "b84e01786241ac7b6d58e954cf2222ce"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.843683 7022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:13.861688 7022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:13.862085 7022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:13.862250 7022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:13.862524 7022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:13.862854 7022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:13.862907 7022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.862950 7022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:13.862977 7022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.868908 7022 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.193:35797
I20260924 13:46:13.868952 7149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.193:35797 every 8 connection(s)
I20260924 13:46:13.869292 7022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/data/info.pb
I20260924 13:46:13.872726 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 7022
I20260924 13:46:13.872812 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-0/wal/instance
I20260924 13:46:13.873024 19671 external_mini_cluster.cc:1428] Setting key 95713c3a1971a11f5ef4c2d5e8571b25
I20260924 13:46:13.874089 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.194:0
--local_ip_for_outbound_sockets=127.19.53.194
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41067
--builtin_ntp_servers=127.19.53.212:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 13:46:13.875013 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.870067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.193:46083 (local address 127.19.53.254:41067)
0924 13:46:13.870118 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:13.870120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:13.870375 (+ 255us) server_negotiation.cc:407] Connection header received
0924 13:46:13.870990 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:13.870991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:13.871006 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:13.871079 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:13.871763 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.872260 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.873441 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.873678 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.873749 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.874794 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:13.874795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:13.874795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:13.874801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:13.874814 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:13.874818 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:13.874866 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:13.874935 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 13:46:13.874956 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:46:13.875303 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.870069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:41067 (local address 127.19.53.193:46083)
0924 13:46:13.870245 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:13.870278 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:13.870875 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:13.871159 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:13.871163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:13.871172 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:13.871666 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.871670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.872358 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.872359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.873358 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.873360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.873771 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.873773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.874010 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.874638 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:46:13.874638 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:13.874678 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:46:13.874708 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:13.874868 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:13.874871 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:13.874875 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:13.874933 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 13:46:13.875044 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260924 13:46:13.875831 7150 heartbeater.cc:347] Connected to a master server at 127.19.53.254:41067
I20260924 13:46:13.875937 7150 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:13.876129 7150 heartbeater.cc:511] Master 127.19.53.254:41067 requested a full tablet report, sending...
I20260924 13:46:13.876711 6962 ts_manager.cc:194] Registered new tserver with Master: 0dd3f41bf9934cd99e29c2859bee90d9 (127.19.53.193:35797)
I20260924 13:46:13.877405 6962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.53.193:46083
W20260924 13:46:13.956997 7153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:13.957217 7153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:13.957252 7153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:13.958447 7153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:13.958508 7153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:13.958549 7153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.194
I20260924 13:46:13.959970 7153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.53.194
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.194
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:13.960222 7153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:13.960512 7153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:13.965430 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:13.965464 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:13.965473 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:13.965621 7153 server_base.cc:1034] running on GCE node
I20260924 13:46:13.965830 7153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:13.966096 7153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:13.967254 7153 hybrid_clock.cc:648] HybridClock initialized: now 1790257573967242 us; error 28 us; skew 500 ppm
I20260924 13:46:13.968492 7153 webserver.cc:460] Webserver started at http://127.19.53.194:39009/ using document root <none> and password file <none>
I20260924 13:46:13.968766 7153 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:13.968837 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:13.968991 7153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:13.970137 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/data/instance:
uuid: "7b5148dd0efa49ef8c37e0434b50237b"
format_stamp: "Formatted at 2026-09-24 13:46:13 on dist-test-slave-tfm9"
server_key: "f8b0bb517a81c076980dde4c6439c4e9"
server_key_iv: "058030e04bf6584f53fe951f5e8b5b6c"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.970515 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance:
uuid: "7b5148dd0efa49ef8c37e0434b50237b"
format_stamp: "Formatted at 2026-09-24 13:46:13 on dist-test-slave-tfm9"
server_key: "f8b0bb517a81c076980dde4c6439c4e9"
server_key_iv: "058030e04bf6584f53fe951f5e8b5b6c"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.971886 7153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:13.972617 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.972776 7153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:13.972872 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/wal
uuid: "7b5148dd0efa49ef8c37e0434b50237b"
format_stamp: "Formatted at 2026-09-24 13:46:13 on dist-test-slave-tfm9"
server_key: "f8b0bb517a81c076980dde4c6439c4e9"
server_key_iv: "058030e04bf6584f53fe951f5e8b5b6c"
server_key_version: "encryptionkey@0"
I20260924 13:46:13.972951 7153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:13.986377 7153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:13.986711 7153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:13.986887 7153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:13.987201 7153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:13.987613 7153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:13.987666 7153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.987708 7153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:13.987738 7153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:13.993425 7153 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.194:44291
I20260924 13:46:13.993477 7280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.194:44291 every 8 connection(s)
I20260924 13:46:13.993829 7153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/data/info.pb
I20260924 13:46:13.999241 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.994641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.194:57721 (local address 127.19.53.254:41067)
0924 13:46:13.994700 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:13.994702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:13.994963 (+ 261us) server_negotiation.cc:407] Connection header received
0924 13:46:13.995533 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:13.995535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:13.995548 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:13.995599 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:13.996275 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.996777 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.997784 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.998063 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.998148 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.998966 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:13.998967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:13.998967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:13.998972 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:13.998987 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:13.998991 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:13.999035 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:13.999135 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 13:46:13.999162 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 13:46:13.999509 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:13.994630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:41067 (local address 127.19.53.194:57721)
0924 13:46:13.994844 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:13.994878 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:13.995411 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:13.995670 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:13.995674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:13.995681 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:13.996174 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.996178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.996898 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.996899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.997680 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:13.997682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:13.998159 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:13.998160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:13.998291 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:13.998821 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:46:13.998821 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:13.998852 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:46:13.998873 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:13.999046 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:13.999050 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:13.999054 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:46:13.999136 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 13:46:13.999234 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":104,"threads_started":1}
I20260924 13:46:13.999887 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 7153
I20260924 13:46:14.000015 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/wal/instance
I20260924 13:46:14.000113 7281 heartbeater.cc:347] Connected to a master server at 127.19.53.254:41067
I20260924 13:46:14.000206 7281 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:14.000322 19671 external_mini_cluster.cc:1428] Setting key d29a917b50abea5cb227f4664e13eec3
I20260924 13:46:14.000409 7281 heartbeater.cc:511] Master 127.19.53.254:41067 requested a full tablet report, sending...
I20260924 13:46:14.000838 6961 ts_manager.cc:194] Registered new tserver with Master: 7b5148dd0efa49ef8c37e0434b50237b (127.19.53.194:44291)
I20260924 13:46:14.001276 6961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.53.194:57721
I20260924 13:46:14.001286 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.53.195:0
--local_ip_for_outbound_sockets=127.19.53.195
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41067
--builtin_ntp_servers=127.19.53.212:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:46:14.082199 7284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:14.082435 7284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:14.082469 7284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:14.083604 7284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:14.083676 7284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:14.083719 7284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.53.195
I20260924 13:46:14.085069 7284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.53.195
--webserver_port=0
--tserver_master_addrs=127.19.53.254:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.53.195
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:14.085335 7284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:14.085631 7284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:46:14.091135 7284 server_base.cc:1034] running on GCE node
W20260924 13:46:14.091100 7289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:14.091106 7290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:14.091085 7292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:14.091527 7284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:14.091753 7284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:14.092928 7284 hybrid_clock.cc:648] HybridClock initialized: now 1790257574092917 us; error 26 us; skew 500 ppm
I20260924 13:46:14.094189 7284 webserver.cc:460] Webserver started at http://127.19.53.195:44025/ using document root <none> and password file <none>
I20260924 13:46:14.094456 7284 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:14.094529 7284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:14.094676 7284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:14.095760 7284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/data/instance:
uuid: "360d593f25b24ac48441aefd0f57c5dd"
format_stamp: "Formatted at 2026-09-24 13:46:14 on dist-test-slave-tfm9"
server_key: "06101135c847de41c4538ea2c7420eb6"
server_key_iv: "8c26dda301ef1df449fe1a05711aebbd"
server_key_version: "encryptionkey@0"
I20260924 13:46:14.096122 7284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance:
uuid: "360d593f25b24ac48441aefd0f57c5dd"
format_stamp: "Formatted at 2026-09-24 13:46:14 on dist-test-slave-tfm9"
server_key: "06101135c847de41c4538ea2c7420eb6"
server_key_iv: "8c26dda301ef1df449fe1a05711aebbd"
server_key_version: "encryptionkey@0"
I20260924 13:46:14.097417 7284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:46:14.098179 7298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:14.098356 7284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:14.098456 7284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/wal
uuid: "360d593f25b24ac48441aefd0f57c5dd"
format_stamp: "Formatted at 2026-09-24 13:46:14 on dist-test-slave-tfm9"
server_key: "06101135c847de41c4538ea2c7420eb6"
server_key_iv: "8c26dda301ef1df449fe1a05711aebbd"
server_key_version: "encryptionkey@0"
I20260924 13:46:14.098533 7284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:14.118810 7284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:14.119153 7284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:14.119308 7284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:14.119544 7284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:46:14.119879 7284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:46:14.119935 7284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:14.119976 7284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:46:14.120003 7284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:14.126013 7284 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.195:35287
I20260924 13:46:14.126096 7411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.195:35287 every 8 connection(s)
I20260924 13:46:14.126484 7284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/data/info.pb
I20260924 13:46:14.132167 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.127386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:40367 (local address 127.19.53.254:41067)
0924 13:46:14.127475 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:14.127477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:14.127576 (+ 99us) server_negotiation.cc:407] Connection header received
0924 13:46:14.128414 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:14.128415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:14.128429 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:14.128478 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:14.129167 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.129677 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.130716 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.130932 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.131018 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.131913 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:14.131914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:14.131915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:14.131919 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:14.131935 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:14.131939 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:14.131982 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:14.132057 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 13:46:14.132089 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 13:46:14.132377 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.127271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:41067 (local address 127.19.53.195:40367)
0924 13:46:14.127459 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:14.127501 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:14.128241 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:14.128560 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:14.128564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:14.128576 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:14.129069 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:14.129073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.129772 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.129773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:14.130621 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:14.130623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.131027 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.131029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:14.131171 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.131764 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:46:14.131765 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:14.131791 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:46:14.131820 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:14.131992 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:14.131997 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:14.132000 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:46:14.132061 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 13:46:14.132159 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260924 13:46:14.132849 7412 heartbeater.cc:347] Connected to a master server at 127.19.53.254:41067
I20260924 13:46:14.132936 7412 heartbeater.cc:464] Registering TS with master...
I20260924 13:46:14.133075 7412 heartbeater.cc:511] Master 127.19.53.254:41067 requested a full tablet report, sending...
I20260924 13:46:14.133467 6961 ts_manager.cc:194] Registered new tserver with Master: 360d593f25b24ac48441aefd0f57c5dd (127.19.53.195:35287)
I20260924 13:46:14.133776 6961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.53.195:40367
I20260924 13:46:14.136215 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 7284
I20260924 13:46:14.136301 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/wal/instance
I20260924 13:46:14.136502 19671 external_mini_cluster.cc:1428] Setting key 2c3a3b1fe26df46bee79a488ed68249c
I20260924 13:46:14.137451 19671 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:46:14.141793 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.138821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.19.53.254:41067)
0924 13:46:14.138895 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:14.138897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:14.138969 (+ 72us) server_negotiation.cc:407] Connection header received
0924 13:46:14.139042 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:14.139043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:14.139055 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:14.139122 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:14.139306 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.139885 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.140655 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.140864 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.140938 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.141551 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:14.141552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:14.141553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:14.141557 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:14.141570 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:14.141574 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:14.141625 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:14.141688 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 13:46:14.141711 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 13:46:14.143316 6961 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:46:14.144820 6961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:46:14.151386 7421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.146908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:44291 (local address 127.0.0.1:39620)
0924 13:46:14.147089 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:14.147118 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:14.147187 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:14.147650 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:14.147653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:14.147656 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:14.147720 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:14.147722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.149083 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.149083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:14.150218 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:14.150221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.151136 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.151137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:14.151188 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.151190 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:46:14.151254 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 13:46:14.151303 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":44,"thread_start_us":66,"threads_started":1}
I20260924 13:46:14.151438 7424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.147280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.19.53.194:44291)
0924 13:46:14.147441 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:14.147447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:14.147461 (+ 14us) server_negotiation.cc:407] Connection header received
0924 13:46:14.147514 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:14.147516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:14.147532 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:14.147798 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:14.147862 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.149000 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.150291 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.151036 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.151128 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.151214 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:14.151266 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 13:46:14.151348 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260924 13:46:14.151309 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.147131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.195:35287 (local address 127.0.0.1:57522)
0924 13:46:14.147709 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:14.147718 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:14.147800 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:14.148019 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:14.148019 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:14.148021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:14.148067 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:14.148069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.148892 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.148893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:14.150218 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:14.150221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.150963 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.150964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:14.151058 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.151073 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 13:46:14.151142 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 13:46:14.151190 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":21,"thread_start_us":71,"threads_started":1}
I20260924 13:46:14.151568 7425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.147501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.19.53.193:35797)
0924 13:46:14.147676 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:14.147681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:14.147691 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:46:14.147727 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:14.147730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:14.147746 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:14.147826 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:14.148002 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.149072 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.150226 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.150953 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.151040 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.151132 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:14.151427 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 13:46:14.151473 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260924 13:46:14.151691 7422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.146921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:35797 (local address 127.0.0.1:39412)
0924 13:46:14.147245 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:14.147256 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:14.147306 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:14.147893 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:14.147893 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:14.147896 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:14.147929 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:14.147930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.149130 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.149131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:14.150146 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:14.150148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.151166 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.151167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:14.151247 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.151252 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:46:14.151331 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 13:46:14.151368 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260924 13:46:14.152077 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.147225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.19.53.195:35287)
0924 13:46:14.147366 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:14.147370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:14.147772 (+ 402us) server_negotiation.cc:407] Connection header received
0924 13:46:14.147907 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:14.147910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:14.147925 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:14.148021 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:14.148120 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.148805 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.150328 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.150857 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.150960 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.151066 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:14.151359 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 13:46:14.151397 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":88,"threads_started":1}
I20260924 13:46:14.152119 7215 tablet_service.cc:1467] Processing CreateTablet for tablet 89007101240e454c82b1e9f8b99fa7d6 (DEFAULT_TABLE table=test-table [id=276f4850b93d413aa27fb4a767405b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:14.152482 7215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89007101240e454c82b1e9f8b99fa7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:14.152395 7084 tablet_service.cc:1467] Processing CreateTablet for tablet 89007101240e454c82b1e9f8b99fa7d6 (DEFAULT_TABLE table=test-table [id=276f4850b93d413aa27fb4a767405b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:14.152709 7084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89007101240e454c82b1e9f8b99fa7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:14.155958 7427 tablet_bootstrap.cc:492] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9: Bootstrap starting.
I20260924 13:46:14.156383 7346 tablet_service.cc:1467] Processing CreateTablet for tablet 89007101240e454c82b1e9f8b99fa7d6 (DEFAULT_TABLE table=test-table [id=276f4850b93d413aa27fb4a767405b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:46:14.156663 7427 tablet_bootstrap.cc:654] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:14.156684 7346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89007101240e454c82b1e9f8b99fa7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:14.157017 7427 log.cc:824] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:14.157375 7426 tablet_bootstrap.cc:492] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b: Bootstrap starting.
I20260924 13:46:14.157789 7427 tablet_bootstrap.cc:492] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9: No bootstrap required, opened a new log
I20260924 13:46:14.157871 7427 ts_tablet_manager.cc:1397] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:46:14.158036 7426 tablet_bootstrap.cc:654] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:14.158370 7426 log.cc:824] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:14.159129 7426 tablet_bootstrap.cc:492] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b: No bootstrap required, opened a new log
I20260924 13:46:14.159199 7426 ts_tablet_manager.cc:1397] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:46:14.159591 7430 tablet_bootstrap.cc:492] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd: Bootstrap starting.
I20260924 13:46:14.160163 7430 tablet_bootstrap.cc:654] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:14.160449 7430 log.cc:824] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:14.161052 7430 tablet_bootstrap.cc:492] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd: No bootstrap required, opened a new log
I20260924 13:46:14.160954 7427 raft_consensus.cc:348] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5148dd0efa49ef8c37e0434b50237b" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44291 } } peers { permanent_uuid: "360d593f25b24ac48441aefd0f57c5dd" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35287 } } peers { permanent_uuid: "0dd3f41bf9934cd99e29c2859bee90d9" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35797 } }
I20260924 13:46:14.161114 7427 raft_consensus.cc:374] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:14.161131 7430 ts_tablet_manager.cc:1397] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:46:14.161156 7427 raft_consensus.cc:729] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd3f41bf9934cd99e29c2859bee90d9, State: Initialized, Role: FOLLOWER
I20260924 13:46:14.161298 7427 consensus_queue.cc:260] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5148dd0efa49ef8c37e0434b50237b" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44291 } } peers { permanent_uuid: "360d593f25b24ac48441aefd0f57c5dd" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35287 } } peers { permanent_uuid: "0dd3f41bf9934cd99e29c2859bee90d9" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35797 } }
I20260924 13:46:14.161564 7427 ts_tablet_manager.cc:1428] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260924 13:46:14.161695 7150 heartbeater.cc:503] Master 127.19.53.254:41067 was elected leader, sending a full tablet report...
I20260924 13:46:14.161859 7426 raft_consensus.cc:348] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5148dd0efa49ef8c37e0434b50237b" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44291 } } peers { permanent_uuid: "360d593f25b24ac48441aefd0f57c5dd" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35287 } } peers { permanent_uuid: "0dd3f41bf9934cd99e29c2859bee90d9" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35797 } }
I20260924 13:46:14.162070 7426 raft_consensus.cc:374] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:14.162110 7426 raft_consensus.cc:729] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5148dd0efa49ef8c37e0434b50237b, State: Initialized, Role: FOLLOWER
I20260924 13:46:14.162218 7426 consensus_queue.cc:260] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5148dd0efa49ef8c37e0434b50237b" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44291 } } peers { permanent_uuid: "360d593f25b24ac48441aefd0f57c5dd" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35287 } } peers { permanent_uuid: "0dd3f41bf9934cd99e29c2859bee90d9" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35797 } }
I20260924 13:46:14.162483 7426 ts_tablet_manager.cc:1428] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 13:46:14.162503 7281 heartbeater.cc:503] Master 127.19.53.254:41067 was elected leader, sending a full tablet report...
I20260924 13:46:14.163779 7430 raft_consensus.cc:348] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5148dd0efa49ef8c37e0434b50237b" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44291 } } peers { permanent_uuid: "360d593f25b24ac48441aefd0f57c5dd" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35287 } } peers { permanent_uuid: "0dd3f41bf9934cd99e29c2859bee90d9" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35797 } }
I20260924 13:46:14.163923 7430 raft_consensus.cc:374] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:14.163967 7430 raft_consensus.cc:729] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360d593f25b24ac48441aefd0f57c5dd, State: Initialized, Role: FOLLOWER
I20260924 13:46:14.164073 7430 consensus_queue.cc:260] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5148dd0efa49ef8c37e0434b50237b" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44291 } } peers { permanent_uuid: "360d593f25b24ac48441aefd0f57c5dd" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35287 } } peers { permanent_uuid: "0dd3f41bf9934cd99e29c2859bee90d9" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35797 } }
I20260924 13:46:14.164319 7430 ts_tablet_manager.cc:1428] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:46:14.164354 7412 heartbeater.cc:503] Master 127.19.53.254:41067 was elected leader, sending a full tablet report...
I20260924 13:46:14.213629 7434 raft_consensus.cc:482] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:46:14.213775 7434 raft_consensus.cc:504] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5148dd0efa49ef8c37e0434b50237b" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44291 } } peers { permanent_uuid: "360d593f25b24ac48441aefd0f57c5dd" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35287 } } peers { permanent_uuid: "0dd3f41bf9934cd99e29c2859bee90d9" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35797 } }
I20260924 13:46:14.214183 7434 leader_election.cc:288] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b5148dd0efa49ef8c37e0434b50237b (127.19.53.194:44291), 0dd3f41bf9934cd99e29c2859bee90d9 (127.19.53.193:35797)
I20260924 13:46:14.217015 7424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.214447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:41835 (local address 127.19.53.194:44291)
0924 13:46:14.214491 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:14.214493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:14.214502 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:46:14.214535 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:14.214536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:14.214545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:14.214607 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:14.214809 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.215340 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.216304 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.216675 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.216731 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.216759 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:14.216897 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 13:46:14.216944 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:46:14.217047 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.214338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.194:44291 (local address 127.19.53.195:41835)
0924 13:46:14.214391 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:14.214409 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:14.214454 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:14.214649 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:14.214650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:14.214652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:14.214736 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:14.214739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.215420 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.215421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:14.216217 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:14.216219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.216753 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.216754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:14.216822 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.216825 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:46:14.216875 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:46:14.216912 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":23}
I20260924 13:46:14.217077 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.214352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:35797 (local address 127.19.53.195:40357)
0924 13:46:14.214566 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:14.214575 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:14.214635 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:14.214786 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:14.214787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:14.214789 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:46:14.214860 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:14.214862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.215485 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.215486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:14.216240 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:14.216241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.216793 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.216794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:14.216883 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.216889 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 13:46:14.216943 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 13:46:14.216975 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":50,"thread_start_us":98,"threads_started":1}
I20260924 13:46:14.217078 7425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.214448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.53.195:40357 (local address 127.19.53.193:35797)
0924 13:46:14.214491 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:14.214493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:14.214647 (+ 154us) server_negotiation.cc:407] Connection header received
0924 13:46:14.214689 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:14.214690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:14.214705 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:14.214756 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:46:14.214933 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.215400 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.216356 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.216688 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.216774 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.216782 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:14.216965 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 13:46:14.217012 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:46:14.217445 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89007101240e454c82b1e9f8b99fa7d6" candidate_uuid: "360d593f25b24ac48441aefd0f57c5dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd3f41bf9934cd99e29c2859bee90d9" is_pre_election: true
I20260924 13:46:14.217609 7104 raft_consensus.cc:2454] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360d593f25b24ac48441aefd0f57c5dd in term 0.
I20260924 13:46:14.217576 7235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89007101240e454c82b1e9f8b99fa7d6" candidate_uuid: "360d593f25b24ac48441aefd0f57c5dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5148dd0efa49ef8c37e0434b50237b" is_pre_election: true
I20260924 13:46:14.217744 7235 raft_consensus.cc:2454] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360d593f25b24ac48441aefd0f57c5dd in term 0.
I20260924 13:46:14.217866 7302 leader_election.cc:302] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dd3f41bf9934cd99e29c2859bee90d9, 360d593f25b24ac48441aefd0f57c5dd; no voters:
I20260924 13:46:14.218077 7434 raft_consensus.cc:2781] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:46:14.218137 7434 raft_consensus.cc:482] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:46:14.218161 7434 raft_consensus.cc:3037] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:14.218827 7434 raft_consensus.cc:504] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5148dd0efa49ef8c37e0434b50237b" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44291 } } peers { permanent_uuid: "360d593f25b24ac48441aefd0f57c5dd" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35287 } } peers { permanent_uuid: "0dd3f41bf9934cd99e29c2859bee90d9" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35797 } }
I20260924 13:46:14.218971 7434 leader_election.cc:288] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [CANDIDATE]: Term 1 election: Requested vote from peers 7b5148dd0efa49ef8c37e0434b50237b (127.19.53.194:44291), 0dd3f41bf9934cd99e29c2859bee90d9 (127.19.53.193:35797)
I20260924 13:46:14.219112 7235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89007101240e454c82b1e9f8b99fa7d6" candidate_uuid: "360d593f25b24ac48441aefd0f57c5dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5148dd0efa49ef8c37e0434b50237b"
I20260924 13:46:14.219139 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89007101240e454c82b1e9f8b99fa7d6" candidate_uuid: "360d593f25b24ac48441aefd0f57c5dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd3f41bf9934cd99e29c2859bee90d9"
I20260924 13:46:14.219200 7104 raft_consensus.cc:3037] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:14.219213 7235 raft_consensus.cc:3037] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:14.219923 7104 raft_consensus.cc:2454] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360d593f25b24ac48441aefd0f57c5dd in term 1.
I20260924 13:46:14.220032 7235 raft_consensus.cc:2454] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360d593f25b24ac48441aefd0f57c5dd in term 1.
I20260924 13:46:14.220085 7302 leader_election.cc:302] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dd3f41bf9934cd99e29c2859bee90d9, 360d593f25b24ac48441aefd0f57c5dd; no voters:
I20260924 13:46:14.220157 7434 raft_consensus.cc:2781] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:14.220289 7434 raft_consensus.cc:686] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [term 1 LEADER]: Becoming Leader. State: Replica: 360d593f25b24ac48441aefd0f57c5dd, State: Running, Role: LEADER
I20260924 13:46:14.220408 7434 consensus_queue.cc:237] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5148dd0efa49ef8c37e0434b50237b" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44291 } } peers { permanent_uuid: "360d593f25b24ac48441aefd0f57c5dd" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35287 } } peers { permanent_uuid: "0dd3f41bf9934cd99e29c2859bee90d9" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35797 } }
I20260924 13:46:14.221325 6962 catalog_manager.cc:5495] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 360d593f25b24ac48441aefd0f57c5dd (127.19.53.195). New cstate: current_term: 1 leader_uuid: "360d593f25b24ac48441aefd0f57c5dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5148dd0efa49ef8c37e0434b50237b" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360d593f25b24ac48441aefd0f57c5dd" member_type: VOTER last_known_addr { host: "127.19.53.195" port: 35287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dd3f41bf9934cd99e29c2859bee90d9" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35797 } health_report { overall_health: UNKNOWN } } }
I20260924 13:46:14.244078 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.241666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57538 (local address 127.19.53.195:35287)
0924 13:46:14.241714 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:14.241716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:14.241725 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:46:14.241798 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:14.241799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:14.241821 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:14.241917 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:14.242073 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.242537 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.243279 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:14.243477 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:14.243560 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:14.243720 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:14.243822 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:14.243924 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:14.243943 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 13:46:14.243964 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 13:46:14.244357 7282 tablet.cc:2369] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:46:14.245733 7235 raft_consensus.cc:1261] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b [term 1 FOLLOWER]: Refusing update from remote peer 360d593f25b24ac48441aefd0f57c5dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:14.245740 7104 raft_consensus.cc:1261] T 89007101240e454c82b1e9f8b99fa7d6 P 0dd3f41bf9934cd99e29c2859bee90d9 [term 1 FOLLOWER]: Refusing update from remote peer 360d593f25b24ac48441aefd0f57c5dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:14.246131 7434 consensus_queue.cc:1035] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dd3f41bf9934cd99e29c2859bee90d9" member_type: VOTER last_known_addr { host: "127.19.53.193" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:14.246256 7436 consensus_queue.cc:1035] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5148dd0efa49ef8c37e0434b50237b" member_type: VOTER last_known_addr { host: "127.19.53.194" port: 44291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:14.248574 7439 mvcc.cc:205] Tried to move back new op lower bound from 7332895024108150784 to 7332895024007639040. Current Snapshot: MvccSnapshot[applied={T|T < 7332895024108150784}]
I20260924 13:46:14.248878 7441 mvcc.cc:205] Tried to move back new op lower bound from 7332895024108150784 to 7332895024007639040. Current Snapshot: MvccSnapshot[applied={T|T < 7332895024108150784}]
I20260924 13:46:14.250962 6962 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:46:14.251072 6962 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:46:14.251871 6962 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8160620f46f1486b894455006f6f5c82: Sending DeleteTablet for 3 replicas of tablet 89007101240e454c82b1e9f8b99fa7d6
I20260924 13:46:14.252221 7084 tablet_service.cc:1514] Processing DeleteTablet for tablet 89007101240e454c82b1e9f8b99fa7d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:14 UTC) from {username='slave'} at 127.0.0.1:39412
I20260924 13:46:14.252242 7215 tablet_service.cc:1514] Processing DeleteTablet for tablet 89007101240e454c82b1e9f8b99fa7d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:14 UTC) from {username='slave'} at 127.0.0.1:39620
I20260924 13:46:14.252257 7346 tablet_service.cc:1514] Processing DeleteTablet for tablet 89007101240e454c82b1e9f8b99fa7d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:46:14 UTC) from {username='slave'} at 127.0.0.1:57522
I20260924 13:46:14.252481 7449 tablet_replica.cc:330] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b: stopping tablet replica
I20260924 13:46:14.252503 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 7022
I20260924 13:46:14.252584 7449 raft_consensus.cc:2229] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:46:14.252681 7449 raft_consensus.cc:2258] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:46:14.252810 7450 tablet_replica.cc:330] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd: stopping tablet replica
I20260924 13:46:14.252887 7450 raft_consensus.cc:2229] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:46:14.253357 7450 raft_consensus.cc:2258] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:46:14.253441 7449 ts_tablet_manager.cc:1905] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:46:14.253806 7450 ts_tablet_manager.cc:1905] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:46:14.255466 7449 ts_tablet_manager.cc:1918] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:46:14.255541 7449 log.cc:1196] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-1/wal/wals/89007101240e454c82b1e9f8b99fa7d6
I20260924 13:46:14.255651 7450 ts_tablet_manager.cc:1918] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:46:14.255766 7450 log.cc:1196] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd: Deleting WAL directory at /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257335418480-19671-0/minicluster-data/ts-2/wal/wals/89007101240e454c82b1e9f8b99fa7d6
I20260924 13:46:14.255785 7449 ts_tablet_manager.cc:1939] T 89007101240e454c82b1e9f8b99fa7d6 P 7b5148dd0efa49ef8c37e0434b50237b: Deleting consensus metadata
I20260924 13:46:14.256114 7450 ts_tablet_manager.cc:1939] T 89007101240e454c82b1e9f8b99fa7d6 P 360d593f25b24ac48441aefd0f57c5dd: Deleting consensus metadata
I20260924 13:46:14.256162 6946 catalog_manager.cc:4841] TS 7b5148dd0efa49ef8c37e0434b50237b (127.19.53.194:44291): tablet 89007101240e454c82b1e9f8b99fa7d6 (table test-table [id=276f4850b93d413aa27fb4a767405b5c]) successfully deleted
I20260924 13:46:14.256590 6946 catalog_manager.cc:4841] TS 360d593f25b24ac48441aefd0f57c5dd (127.19.53.195:35287): tablet 89007101240e454c82b1e9f8b99fa7d6 (table test-table [id=276f4850b93d413aa27fb4a767405b5c]) successfully deleted
W20260924 13:46:14.260273 6949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.53.193:35797 (error 108)
I20260924 13:46:14.260465 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 7153
I20260924 13:46:14.261267 7422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:14.261091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.193:35797 (local address 127.0.0.1:39426)
0924 13:46:14.261193 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:14.261241 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.193:35797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260924 13:46:14.261391 6949 catalog_manager.cc:4584] TS 0dd3f41bf9934cd99e29c2859bee90d9 (127.19.53.193:35797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89007101240e454c82b1e9f8b99fa7d6: Network error: Client connection negotiation failed: client connection to 127.19.53.193:35797: connect: Connection refused (error 111)
I20260924 13:46:14.265285 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 7284
I20260924 13:46:14.270015 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 6931
2026-09-24T13:46:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (741 ms)
[----------] 6 tests from EncryptionPolicyTest (8502 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:14 dist-test-slave-tfm9 krb5kdc[7455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:14 dist-test-slave-tfm9 krb5kdc[7455](info): set up 2 sockets
Sep 24 13:46:14 dist-test-slave-tfm9 krb5kdc[7455](info): commencing operation
krb5kdc: starting...
W20260924 13:46:15.383538 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:15 dist-test-slave-tfm9 krb5kdc[7455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:46:15.500581 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:15.583073 7468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:15.583293 7468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:15.583318 7468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:15.584573 7468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:15.584628 7468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:15.584643 7468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:15.584657 7468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:15.584669 7468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:15.586196 7468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33523
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:15.586387 7468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:15.586542 7468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:15.591182 7474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:15.591173 7473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:15.591300 7468 server_base.cc:1034] running on GCE node
W20260924 13:46:15.591174 7476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:15.591753 7468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:15.592069 7468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:15.593237 7468 hybrid_clock.cc:648] HybridClock initialized: now 1790257575593219 us; error 32 us; skew 500 ppm
Sep 24 13:46:15 dist-test-slave-tfm9 krb5kdc[7455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:15.595808 7468 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:46:15.596228 7468 webserver.cc:460] Webserver started at http://127.0.0.1:33781/ using document root <none> and password file <none>
I20260924 13:46:15.596403 7468 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:15.596441 7468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:15.596524 7468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:15.597479 7468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "afedf413f4034e859b57127dc8bef449"
format_stamp: "Formatted at 2026-09-24 13:46:15 on dist-test-slave-tfm9"
server_key: "70aff5c29dfe462336f4db651dffcace"
server_key_iv: "ccec548190096a8f68648e72758b0be7"
server_key_version: "encryptionkey@0"
I20260924 13:46:15.597775 7468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "afedf413f4034e859b57127dc8bef449"
format_stamp: "Formatted at 2026-09-24 13:46:15 on dist-test-slave-tfm9"
server_key: "70aff5c29dfe462336f4db651dffcace"
server_key_iv: "ccec548190096a8f68648e72758b0be7"
server_key_version: "encryptionkey@0"
I20260924 13:46:15.599161 7468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:46:15.599980 7483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:15.600170 7468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:46:15.600260 7468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "afedf413f4034e859b57127dc8bef449"
format_stamp: "Formatted at 2026-09-24 13:46:15 on dist-test-slave-tfm9"
server_key: "70aff5c29dfe462336f4db651dffcace"
server_key_iv: "ccec548190096a8f68648e72758b0be7"
server_key_version: "encryptionkey@0"
I20260924 13:46:15.600314 7468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:15.614522 7468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:15.616711 7468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:15.616820 7468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:15.620786 7468 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33523
I20260924 13:46:15.620844 7535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33523 every 8 connection(s)
I20260924 13:46:15.621176 7468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:15.622141 7536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:15.624366 7536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449: Bootstrap starting.
I20260924 13:46:15.624931 7536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:15.625149 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 7468
I20260924 13:46:15.625250 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:15.625273 7536 log.cc:824] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:15.625478 19671 external_mini_cluster.cc:1428] Setting key 5a85dfe8b7d46c091cdef14f37d5e0e4
I20260924 13:46:15.626102 7536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449: No bootstrap required, opened a new log
I20260924 13:46:15.628686 7536 raft_consensus.cc:348] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afedf413f4034e859b57127dc8bef449" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33523 } }
I20260924 13:46:15.628870 7536 raft_consensus.cc:374] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:15.628908 7536 raft_consensus.cc:729] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afedf413f4034e859b57127dc8bef449, State: Initialized, Role: FOLLOWER
I20260924 13:46:15.629026 7536 consensus_queue.cc:260] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afedf413f4034e859b57127dc8bef449" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33523 } }
I20260924 13:46:15.629099 7536 raft_consensus.cc:388] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:15.629127 7536 raft_consensus.cc:482] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:15.629168 7536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:15.629820 7536 raft_consensus.cc:504] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afedf413f4034e859b57127dc8bef449" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33523 } }
I20260924 13:46:15.629971 7536 leader_election.cc:302] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afedf413f4034e859b57127dc8bef449; no voters:
I20260924 13:46:15.630174 7536 leader_election.cc:288] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:15.630213 7541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:15.630461 7541 raft_consensus.cc:686] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [term 1 LEADER]: Becoming Leader. State: Replica: afedf413f4034e859b57127dc8bef449, State: Running, Role: LEADER
I20260924 13:46:15.630529 7536 sys_catalog.cc:564] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:15.630611 7541 consensus_queue.cc:237] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afedf413f4034e859b57127dc8bef449" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33523 } }
I20260924 13:46:15.630997 7542 sys_catalog.cc:455] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afedf413f4034e859b57127dc8bef449. Latest consensus state: current_term: 1 leader_uuid: "afedf413f4034e859b57127dc8bef449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afedf413f4034e859b57127dc8bef449" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33523 } } }
I20260924 13:46:15.631166 7542 sys_catalog.cc:458] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:15.631839 7543 sys_catalog.cc:455] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afedf413f4034e859b57127dc8bef449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afedf413f4034e859b57127dc8bef449" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33523 } } }
I20260924 13:46:15.631922 7543 sys_catalog.cc:458] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:15.632036 7549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 13:46:15 dist-test-slave-tfm9 krb5kdc[7455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 13:46:15.632712 7549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:15.634245 7549 catalog_manager.cc:1347] Generated new cluster ID: b5d766664888418198f71ca53ba13f8b
I20260924 13:46:15.634299 7549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:15.636894 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:15.626022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59983 (local address 127.0.0.1:33523)
0924 13:46:15.626182 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:15.626197 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:15.626222 (+ 25us) server_negotiation.cc:407] Connection header received
0924 13:46:15.626657 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:15.626664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:15.626716 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:15.626845 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:15.627624 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:15.628477 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:15.629543 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:15.630053 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:15.630194 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:15.633154 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:15.633158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:15.633158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:15.633171 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:15.635214 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:15.635651 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:15.635654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:15.635655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:15.635760 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:15.636007 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:15.636008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:15.636009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:15.636303 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:15.636469 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:15.636551 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 13:46:15.636652 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260924 13:46:15.637930 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:15.642098 7549 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:15.642733 7549 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:15.646809 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:15.639526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40099 (local address 127.0.0.1:33523)
0924 13:46:15.639555 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:15.639557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:15.639578 (+ 21us) server_negotiation.cc:407] Connection header received
0924 13:46:15.639654 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:15.639654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:15.639663 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:15.639706 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:15.640384 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:15.641096 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:15.642157 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:15.642486 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:15.642554 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:15.643908 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:15.643909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:15.643909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:15.643916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:15.645469 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:15.645933 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:15.645935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:15.645935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:15.645988 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:15.646216 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:15.646217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:15.646217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:15.646334 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:15.646420 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:15.646583 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 13:46:15.646641 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 13:46:15.651506 7549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afedf413f4034e859b57127dc8bef449: Generated new TSK 0
I20260924 13:46:15.651719 7549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:15.654501 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 7468
W20260924 13:46:15.658814 7453 connection.cc:535] client connection to 127.0.0.1:33523 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33523): Connection reset by peer (error 104)
2026-09-24T13:46:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:15 dist-test-slave-tfm9 krb5kdc[7567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:15 dist-test-slave-tfm9 krb5kdc[7567](info): set up 2 sockets
Sep 24 13:46:15 dist-test-slave-tfm9 krb5kdc[7567](info): commencing operation
krb5kdc: starting...
W20260924 13:46:16.687867 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:16 dist-test-slave-tfm9 krb5kdc[7567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:46:16.804371 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:16.882325 7580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:16.882598 7580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:16.882637 7580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:16.883788 7580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:16.883864 7580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:16.883889 7580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:16.883916 7580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:16.883945 7580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:16.885432 7580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:16.885704 7580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:16.885955 7580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:16.890762 7586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:16.890750 7585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:16.890976 7580 server_base.cc:1034] running on GCE node
W20260924 13:46:16.890750 7588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:16.891273 7580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:16.891634 7580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:16.892786 7580 hybrid_clock.cc:648] HybridClock initialized: now 1790257576892773 us; error 43 us; skew 500 ppm
Sep 24 13:46:16 dist-test-slave-tfm9 krb5kdc[7567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:16.902858 7580 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:46:16.903470 7580 webserver.cc:460] Webserver started at http://127.0.0.1:43949/ using document root <none> and password file <none>
I20260924 13:46:16.903724 7580 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:16.903775 7580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:16.903956 7580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:16.905098 7580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "bd25ea2189f44e78a7442f49ad535f34"
format_stamp: "Formatted at 2026-09-24 13:46:16 on dist-test-slave-tfm9"
server_key: "32df4bf69ad3ddadb6e2f4a2569fc0a8"
server_key_iv: "0c8d396b10f1a8adba86f295bf4d0c6f"
server_key_version: "encryptionkey@0"
I20260924 13:46:16.905609 7580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "bd25ea2189f44e78a7442f49ad535f34"
format_stamp: "Formatted at 2026-09-24 13:46:16 on dist-test-slave-tfm9"
server_key: "32df4bf69ad3ddadb6e2f4a2569fc0a8"
server_key_iv: "0c8d396b10f1a8adba86f295bf4d0c6f"
server_key_version: "encryptionkey@0"
I20260924 13:46:16.907119 7580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:46:16.907935 7595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:16.908195 7580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:16.908269 7580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "bd25ea2189f44e78a7442f49ad535f34"
format_stamp: "Formatted at 2026-09-24 13:46:16 on dist-test-slave-tfm9"
server_key: "32df4bf69ad3ddadb6e2f4a2569fc0a8"
server_key_iv: "0c8d396b10f1a8adba86f295bf4d0c6f"
server_key_version: "encryptionkey@0"
I20260924 13:46:16.908367 7580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:16.920145 7580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:16.921999 7580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:16.922163 7580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:16.926276 7580 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46477
I20260924 13:46:16.926381 7647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46477 every 8 connection(s)
I20260924 13:46:16.926748 7580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:16.927825 7648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:16.928505 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 7580
I20260924 13:46:16.928622 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:16.928830 19671 external_mini_cluster.cc:1428] Setting key 18f561dcb0f9f7879cc8de887cb5ea82
I20260924 13:46:16.930694 7648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34: Bootstrap starting.
I20260924 13:46:16.931286 7648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:16.931576 7648 log.cc:824] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:16.932363 7648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34: No bootstrap required, opened a new log
I20260924 13:46:16.934361 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:16.929346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40577 (local address 127.0.0.1:46477)
0924 13:46:16.929531 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:16.929543 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:16.929573 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:46:16.930057 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:16.930064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:16.930114 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:16.930259 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:16.930452 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:16.931401 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:16.932389 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:16.932856 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:16.932975 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:16.933724 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:16.933729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:16.933729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:16.933891 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:16.933981 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:16.933993 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:16.934082 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:16.934161 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 13:46:16.934190 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260924 13:46:16.934724 7648 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd25ea2189f44e78a7442f49ad535f34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } }
I20260924 13:46:16.934893 7648 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:16.934935 7648 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd25ea2189f44e78a7442f49ad535f34, State: Initialized, Role: FOLLOWER
I20260924 13:46:16.935092 7648 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd25ea2189f44e78a7442f49ad535f34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } }
I20260924 13:46:16.935199 7648 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:16.935298 7648 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:16.935362 7648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:16.935467 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:16.935993 7648 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd25ea2189f44e78a7442f49ad535f34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } }
I20260924 13:46:16.936085 7648 leader_election.cc:302] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd25ea2189f44e78a7442f49ad535f34; no voters:
I20260924 13:46:16.936337 7648 leader_election.cc:288] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:16.936368 7653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:16.936568 7653 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [term 1 LEADER]: Becoming Leader. State: Replica: bd25ea2189f44e78a7442f49ad535f34, State: Running, Role: LEADER
I20260924 13:46:16.936672 7648 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:16.936697 7653 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd25ea2189f44e78a7442f49ad535f34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } }
I20260924 13:46:16.937052 7653 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd25ea2189f44e78a7442f49ad535f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd25ea2189f44e78a7442f49ad535f34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } } }
I20260924 13:46:16.937134 7653 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:16.937306 7653 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd25ea2189f44e78a7442f49ad535f34. Latest consensus state: current_term: 1 leader_uuid: "bd25ea2189f44e78a7442f49ad535f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd25ea2189f44e78a7442f49ad535f34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } } }
I20260924 13:46:16.937446 7653 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:16.937460 7663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:16.938057 7663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:16.940284 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:16.937050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46327 (local address 127.0.0.1:46477)
0924 13:46:16.937088 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:16.937091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:16.937097 (+ 6us) server_negotiation.cc:407] Connection header received
0924 13:46:16.937115 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:16.937116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:16.937124 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:16.937170 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:16.937363 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:16.938104 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:16.939115 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:16.939335 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:16.939394 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:16.940069 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:16.940071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:16.940071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:16.940075 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:16.940088 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:16.940092 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:16.940140 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:16.940225 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 13:46:16.940235 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:46:16.941067 7663 catalog_manager.cc:1347] Generated new cluster ID: 4b2943901f4b419db77833424683558a
I20260924 13:46:16.941147 7663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:16.959645 7663 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:16.960189 7663 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:16.965109 7663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd25ea2189f44e78a7442f49ad535f34: Generated new TSK 0
I20260924 13:46:16.965353 7663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:16.968014 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 7580
W20260924 13:46:16.972200 7565 connection.cc:535] client connection to 127.0.0.1:46477 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46477): Connection reset by peer (error 104)
2026-09-24T13:46:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:17 dist-test-slave-tfm9 krb5kdc[7679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:17 dist-test-slave-tfm9 krb5kdc[7679](info): set up 2 sockets
Sep 24 13:46:17 dist-test-slave-tfm9 krb5kdc[7679](info): commencing operation
krb5kdc: starting...
W20260924 13:46:18.007506 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:18 dist-test-slave-tfm9 krb5kdc[7679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:46:18.122674 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:18.202885 7692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:18.203118 7692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:18.203157 7692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:18.204291 7692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:18.204366 7692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:18.204391 7692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:18.204420 7692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:18.204450 7692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:18.205937 7692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:18.206211 7692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:18.206447 7692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:18.211105 7698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:18.211273 7692 server_base.cc:1034] running on GCE node
W20260924 13:46:18.211084 7697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:18.211084 7700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:18.211658 7692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:18.211951 7692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:18.213117 7692 hybrid_clock.cc:648] HybridClock initialized: now 1790257578213103 us; error 44 us; skew 500 ppm
Sep 24 13:46:18 dist-test-slave-tfm9 krb5kdc[7679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:18.215664 7692 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:46:18.216084 7692 webserver.cc:460] Webserver started at http://127.0.0.1:34397/ using document root <none> and password file <none>
I20260924 13:46:18.216305 7692 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:18.216347 7692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:18.216534 7692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:18.217638 7692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "9554093fe6b9488c9cc7ad636f4bc623"
format_stamp: "Formatted at 2026-09-24 13:46:18 on dist-test-slave-tfm9"
server_key: "1d85f9b6b60b54c3499cb0ce5c529160"
server_key_iv: "2e556e4e423aabf03de6ed546b6f0544"
server_key_version: "encryptionkey@0"
I20260924 13:46:18.218050 7692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "9554093fe6b9488c9cc7ad636f4bc623"
format_stamp: "Formatted at 2026-09-24 13:46:18 on dist-test-slave-tfm9"
server_key: "1d85f9b6b60b54c3499cb0ce5c529160"
server_key_iv: "2e556e4e423aabf03de6ed546b6f0544"
server_key_version: "encryptionkey@0"
I20260924 13:46:18.219364 7692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:18.220158 7707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:18.220352 7692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:18.220436 7692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "9554093fe6b9488c9cc7ad636f4bc623"
format_stamp: "Formatted at 2026-09-24 13:46:18 on dist-test-slave-tfm9"
server_key: "1d85f9b6b60b54c3499cb0ce5c529160"
server_key_iv: "2e556e4e423aabf03de6ed546b6f0544"
server_key_version: "encryptionkey@0"
I20260924 13:46:18.220490 7692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:18.224730 7692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:18.227711 7692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:18.227851 7692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:18.232651 7692 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40729
I20260924 13:46:18.232903 7759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40729 every 8 connection(s)
I20260924 13:46:18.233044 7692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:18.233950 7760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:18.237141 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 7692
I20260924 13:46:18.237250 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:18.237372 7760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623: Bootstrap starting.
I20260924 13:46:18.237459 19671 external_mini_cluster.cc:1428] Setting key 37afd39c9c217ee963b69ae47678bb4a
I20260924 13:46:18.238157 7760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:18.238617 7760 log.cc:824] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:18.239318 7760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623: No bootstrap required, opened a new log
I20260924 13:46:18.241034 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:18.237972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44473 (local address 127.0.0.1:40729)
0924 13:46:18.238199 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:18.238208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:18.238853 (+ 645us) server_negotiation.cc:407] Connection header received
0924 13:46:18.239266 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:18.239272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:18.239294 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:18.239454 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:18.240434 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:18.240438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:18.240438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:18.240443 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:18.240486 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:18.240499 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:18.240584 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:18.240792 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 13:46:18.240834 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
I20260924 13:46:18.242067 7760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9554093fe6b9488c9cc7ad636f4bc623" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40729 } }
I20260924 13:46:18.242229 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:18.242233 7760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:18.242282 7760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9554093fe6b9488c9cc7ad636f4bc623, State: Initialized, Role: FOLLOWER
I20260924 13:46:18.242388 7760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9554093fe6b9488c9cc7ad636f4bc623" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40729 } }
I20260924 13:46:18.242455 7760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:18.242475 7760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:18.242501 7760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:18.242977 7760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9554093fe6b9488c9cc7ad636f4bc623" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40729 } }
I20260924 13:46:18.243043 7760 leader_election.cc:302] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9554093fe6b9488c9cc7ad636f4bc623; no voters:
I20260924 13:46:18.243176 7760 leader_election.cc:288] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:18.243197 7765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:18.243320 7765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [term 1 LEADER]: Becoming Leader. State: Replica: 9554093fe6b9488c9cc7ad636f4bc623, State: Running, Role: LEADER
I20260924 13:46:18.243407 7765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9554093fe6b9488c9cc7ad636f4bc623" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40729 } }
I20260924 13:46:18.243444 7760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:18.243687 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9554093fe6b9488c9cc7ad636f4bc623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9554093fe6b9488c9cc7ad636f4bc623" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40729 } } }
I20260924 13:46:18.243744 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:18.243903 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9554093fe6b9488c9cc7ad636f4bc623. Latest consensus state: current_term: 1 leader_uuid: "9554093fe6b9488c9cc7ad636f4bc623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9554093fe6b9488c9cc7ad636f4bc623" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40729 } } }
I20260924 13:46:18.243999 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623 [sys.catalog]: This master's current role is: LEADER
W20260924 13:46:18.245333 7785 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:46:18.245394 7785 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:46:18.245447 7773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:18.245635 7773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:18.246276 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:18.244822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47477 (local address 127.0.0.1:40729)
0924 13:46:18.244858 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:18.244859 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:18.245230 (+ 371us) server_negotiation.cc:407] Connection header received
0924 13:46:18.245278 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:18.245279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:18.245284 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:18.245346 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:18.246042 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:18.246043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:18.246044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:18.246048 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:18.246062 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:18.246066 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:18.246096 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:18.246198 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 13:46:18.246210 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:46:18.247488 7773 catalog_manager.cc:1347] Generated new cluster ID: 999bff0b1d3f44c7b883f894bdfb3bfb
I20260924 13:46:18.247539 7773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:18.262019 7773 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:18.262208 7773 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:18.270778 7773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9554093fe6b9488c9cc7ad636f4bc623: Generated new TSK 0
I20260924 13:46:18.271059 7773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:18.273839 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 7692
2026-09-24T13:46:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:18 dist-test-slave-tfm9 krb5kdc[7791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:18 dist-test-slave-tfm9 krb5kdc[7791](info): set up 2 sockets
Sep 24 13:46:18 dist-test-slave-tfm9 krb5kdc[7791](info): commencing operation
krb5kdc: starting...
W20260924 13:46:19.299618 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:19 dist-test-slave-tfm9 krb5kdc[7791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:46:19.415012 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:19.494544 7804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:19.494740 7804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:19.494765 7804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:19.495910 7804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:19.495954 7804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:19.495968 7804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:19.496016 7804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:19.496034 7804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:19.497525 7804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:19.497759 7804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:19.498023 7804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:19.502801 7810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:19.502801 7809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:19.502914 7804 server_base.cc:1034] running on GCE node
W20260924 13:46:19.502888 7812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:19.503281 7804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:19.503580 7804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:19.504748 7804 hybrid_clock.cc:648] HybridClock initialized: now 1790257579504743 us; error 37 us; skew 500 ppm
Sep 24 13:46:19 dist-test-slave-tfm9 krb5kdc[7791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:19.507328 7804 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:46:19.507745 7804 webserver.cc:460] Webserver started at http://127.0.0.1:38449/ using document root <none> and password file <none>
I20260924 13:46:19.507974 7804 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:19.508044 7804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:19.508193 7804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:19.509277 7804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "28c7216b1f204d768b7cf32a8a2e007a"
format_stamp: "Formatted at 2026-09-24 13:46:19 on dist-test-slave-tfm9"
server_key: "ce1e978e6699308a161ea88b24d8cbb0"
server_key_iv: "531e4e0942182d3d4f06b5fa62dbcfe9"
server_key_version: "encryptionkey@0"
I20260924 13:46:19.509635 7804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "28c7216b1f204d768b7cf32a8a2e007a"
format_stamp: "Formatted at 2026-09-24 13:46:19 on dist-test-slave-tfm9"
server_key: "ce1e978e6699308a161ea88b24d8cbb0"
server_key_iv: "531e4e0942182d3d4f06b5fa62dbcfe9"
server_key_version: "encryptionkey@0"
I20260924 13:46:19.511020 7804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:46:19.511750 7819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:19.511925 7804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:19.512023 7804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "28c7216b1f204d768b7cf32a8a2e007a"
format_stamp: "Formatted at 2026-09-24 13:46:19 on dist-test-slave-tfm9"
server_key: "ce1e978e6699308a161ea88b24d8cbb0"
server_key_iv: "531e4e0942182d3d4f06b5fa62dbcfe9"
server_key_version: "encryptionkey@0"
I20260924 13:46:19.512104 7804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:19.520686 7804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:19.522949 7804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:19.523102 7804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:19.527040 7804 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35477
I20260924 13:46:19.527083 7871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35477 every 8 connection(s)
I20260924 13:46:19.527467 7804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:19.528314 7872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:19.529079 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 7804
I20260924 13:46:19.529170 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:19.529385 19671 external_mini_cluster.cc:1428] Setting key e434bda44cb31aa03c3482a10ef2e19a
I20260924 13:46:19.531497 7872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a: Bootstrap starting.
I20260924 13:46:19.532065 7872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:19.532348 7872 log.cc:824] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:19.533056 7872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a: No bootstrap required, opened a new log
I20260924 13:46:19.535627 7872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c7216b1f204d768b7cf32a8a2e007a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } }
I20260924 13:46:19.535753 7872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:19.535774 7872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c7216b1f204d768b7cf32a8a2e007a, State: Initialized, Role: FOLLOWER
I20260924 13:46:19.535920 7872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c7216b1f204d768b7cf32a8a2e007a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } }
I20260924 13:46:19.535985 7872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:19.536005 7872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:19.536083 7872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [term 0 FOLLOWER]: Advancing to term 1
Sep 24 13:46:19 dist-test-slave-tfm9 krb5kdc[7791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 13:46:19.536659 7872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c7216b1f204d768b7cf32a8a2e007a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } }
I20260924 13:46:19.536763 7872 leader_election.cc:302] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28c7216b1f204d768b7cf32a8a2e007a; no voters:
I20260924 13:46:19.536936 7872 leader_election.cc:288] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:19.536963 7877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:19.537110 7877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [term 1 LEADER]: Becoming Leader. State: Replica: 28c7216b1f204d768b7cf32a8a2e007a, State: Running, Role: LEADER
I20260924 13:46:19.537199 7877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c7216b1f204d768b7cf32a8a2e007a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } }
I20260924 13:46:19.537225 7872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:19.538491 7878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28c7216b1f204d768b7cf32a8a2e007a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c7216b1f204d768b7cf32a8a2e007a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } } }
I20260924 13:46:19.538594 7878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:19.538692 7879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28c7216b1f204d768b7cf32a8a2e007a. Latest consensus state: current_term: 1 leader_uuid: "28c7216b1f204d768b7cf32a8a2e007a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c7216b1f204d768b7cf32a8a2e007a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } } }
I20260924 13:46:19.538928 7879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a [sys.catalog]: This master's current role is: LEADER
W20260924 13:46:19.539073 7891 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:46:19.539135 7891 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:46:19.540406 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:19.529844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48075 (local address 127.0.0.1:35477)
0924 13:46:19.530047 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:19.530055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:19.530106 (+ 51us) server_negotiation.cc:407] Connection header received
0924 13:46:19.530510 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:19.530516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:19.530558 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:19.530666 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:19.531461 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:19.532396 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:19.533421 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:19.533799 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:19.533888 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:19.536673 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:19.536677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:19.536677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:19.536689 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:19.538861 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:19.539253 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:19.539257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:19.539258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:19.539340 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:19.539573 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:19.539574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:19.539574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:19.539803 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:19.539963 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:19.540069 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 13:46:19.540202 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":104,"threads_started":1}
I20260924 13:46:19.541287 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:19.542048 7893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:19.542263 7893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:19.543902 7893 catalog_manager.cc:1347] Generated new cluster ID: e241eb837c0e4eba813d495a44212b08
I20260924 13:46:19.543951 7893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:19.548977 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:19.542714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60211 (local address 127.0.0.1:35477)
0924 13:46:19.542751 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:19.542753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:19.542867 (+ 114us) server_negotiation.cc:407] Connection header received
0924 13:46:19.542936 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:19.542937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:19.542946 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:19.542998 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:19.543645 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:19.544246 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:19.544936 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:19.545156 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:19.545230 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:19.546458 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:19.546459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:19.546459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:19.546466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:19.547863 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:19.548217 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:19.548218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:19.548219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:19.548269 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:19.548474 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:19.548475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:19.548475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:19.548581 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:19.548677 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:19.548802 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 13:46:19.548876 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:46:19.577317 7893 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:19.577905 7893 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:19.589529 7893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28c7216b1f204d768b7cf32a8a2e007a: Generated new TSK 0
I20260924 13:46:19.589747 7893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:19.592407 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 7804
2026-09-24T13:46:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:19 dist-test-slave-tfm9 krb5kdc[7903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:19 dist-test-slave-tfm9 krb5kdc[7903](info): set up 2 sockets
Sep 24 13:46:19 dist-test-slave-tfm9 krb5kdc[7903](info): commencing operation
krb5kdc: starting...
W20260924 13:46:20.619553 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:20 dist-test-slave-tfm9 krb5kdc[7903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:46:20.734668 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:20.816658 7916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:20.816874 7916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:20.816900 7916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:20.818061 7916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:20.818105 7916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:20.818120 7916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:20.818163 7916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:20.818202 7916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:20.819693 7916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:20.819944 7916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:20.820180 7916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:20.825214 7924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:20.825353 7916 server_base.cc:1034] running on GCE node
W20260924 13:46:20.825234 7921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:20.825392 7922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:20.825711 7916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:20.826102 7916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:20.827242 7916 hybrid_clock.cc:648] HybridClock initialized: now 1790257580827246 us; error 45 us; skew 500 ppm
Sep 24 13:46:20 dist-test-slave-tfm9 krb5kdc[7903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:20.829759 7916 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:46:20.830277 7916 webserver.cc:460] Webserver started at http://127.0.0.1:45571/ using document root <none> and password file <none>
I20260924 13:46:20.830503 7916 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:20.830556 7916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:20.830682 7916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:20.831717 7916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "af374ed6d30c42fc96ea915974cd2f40"
format_stamp: "Formatted at 2026-09-24 13:46:20 on dist-test-slave-tfm9"
server_key: "01f3bcd4af331807f961851f4dda895a"
server_key_iv: "0c3363ad73525a8c982c0126a0d410d8"
server_key_version: "encryptionkey@0"
I20260924 13:46:20.832082 7916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "af374ed6d30c42fc96ea915974cd2f40"
format_stamp: "Formatted at 2026-09-24 13:46:20 on dist-test-slave-tfm9"
server_key: "01f3bcd4af331807f961851f4dda895a"
server_key_iv: "0c3363ad73525a8c982c0126a0d410d8"
server_key_version: "encryptionkey@0"
I20260924 13:46:20.833412 7916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:46:20.834260 7931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:20.834460 7916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:20.834553 7916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "af374ed6d30c42fc96ea915974cd2f40"
format_stamp: "Formatted at 2026-09-24 13:46:20 on dist-test-slave-tfm9"
server_key: "01f3bcd4af331807f961851f4dda895a"
server_key_iv: "0c3363ad73525a8c982c0126a0d410d8"
server_key_version: "encryptionkey@0"
I20260924 13:46:20.834628 7916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:20.845562 7916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:20.847947 7916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:20.848097 7916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:20.852191 7916 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37633
I20260924 13:46:20.852233 7983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37633 every 8 connection(s)
I20260924 13:46:20.852566 7916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:20.853426 7984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:20.855845 7984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40: Bootstrap starting.
I20260924 13:46:20.856390 7984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:20.856779 7984 log.cc:824] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:20.857445 7984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40: No bootstrap required, opened a new log
I20260924 13:46:20.859285 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 7916
I20260924 13:46:20.859241 7984 raft_consensus.cc:348] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af374ed6d30c42fc96ea915974cd2f40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37633 } }
I20260924 13:46:20.859416 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:20.859418 7984 raft_consensus.cc:374] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:20.859484 7984 raft_consensus.cc:729] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af374ed6d30c42fc96ea915974cd2f40, State: Initialized, Role: FOLLOWER
I20260924 13:46:20.859642 19671 external_mini_cluster.cc:1428] Setting key 2bd996fe8519322dd34baf3567f0a370
I20260924 13:46:20.859643 7984 consensus_queue.cc:260] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af374ed6d30c42fc96ea915974cd2f40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37633 } }
I20260924 13:46:20.859776 7984 raft_consensus.cc:388] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:20.859820 7984 raft_consensus.cc:482] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:20.859884 7984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:20.860628 7984 raft_consensus.cc:504] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af374ed6d30c42fc96ea915974cd2f40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37633 } }
I20260924 13:46:20.860752 7984 leader_election.cc:302] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af374ed6d30c42fc96ea915974cd2f40; no voters:
I20260924 13:46:20.860913 7984 leader_election.cc:288] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:20.860972 7989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:20.861114 7989 raft_consensus.cc:686] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [term 1 LEADER]: Becoming Leader. State: Replica: af374ed6d30c42fc96ea915974cd2f40, State: Running, Role: LEADER
I20260924 13:46:20.861169 7984 sys_catalog.cc:564] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:20.861228 7989 consensus_queue.cc:237] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af374ed6d30c42fc96ea915974cd2f40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37633 } }
I20260924 13:46:20.861573 7989 sys_catalog.cc:455] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af374ed6d30c42fc96ea915974cd2f40. Latest consensus state: current_term: 1 leader_uuid: "af374ed6d30c42fc96ea915974cd2f40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af374ed6d30c42fc96ea915974cd2f40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37633 } } }
I20260924 13:46:20.861582 7990 sys_catalog.cc:455] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af374ed6d30c42fc96ea915974cd2f40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af374ed6d30c42fc96ea915974cd2f40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37633 } } }
I20260924 13:46:20.861689 7989 sys_catalog.cc:458] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:20.861760 7990 sys_catalog.cc:458] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:20.862026 7996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:20.862646 7996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:20.864198 7996 catalog_manager.cc:1347] Generated new cluster ID: ffcf68bd3d664a4abd758c829d812222
I20260924 13:46:20.864248 7996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:20.866360 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:20.860204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42467 (local address 127.0.0.1:37633)
0924 13:46:20.860719 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:20.860729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:20.860764 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:46:20.860896 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:20.860904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:20.860961 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:20.861434 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:20.861527 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:20.863060 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:20.864202 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:20.864779 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:20.864900 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:20.865589 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:20.865593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:20.865593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:20.865808 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:20.865848 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:20.865865 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:20.865985 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:20.866059 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 13:46:20.866126 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":388,"threads_started":1}
I20260924 13:46:20.867408 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:20.870991 7996 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:20.871696 7996 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:20.872252 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:20.868836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42939 (local address 127.0.0.1:37633)
0924 13:46:20.868893 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:20.868895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:20.869009 (+ 114us) server_negotiation.cc:407] Connection header received
0924 13:46:20.869041 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:20.869042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:20.869053 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:20.869130 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:20.869263 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:20.870021 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:20.871018 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:20.871233 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:20.871317 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:20.871983 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:20.871984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:20.871985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:20.871990 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:20.872004 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:20.872008 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:20.872055 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:20.872111 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 13:46:20.872147 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":35}
I20260924 13:46:20.879294 7996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af374ed6d30c42fc96ea915974cd2f40: Generated new TSK 0
I20260924 13:46:20.879477 7996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:20.884291 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 7916
2026-09-24T13:46:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:20 dist-test-slave-tfm9 krb5kdc[8015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:20 dist-test-slave-tfm9 krb5kdc[8015](info): set up 2 sockets
Sep 24 13:46:20 dist-test-slave-tfm9 krb5kdc[8015](info): commencing operation
krb5kdc: starting...
W20260924 13:46:21.972227 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:22 dist-test-slave-tfm9 krb5kdc[8015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:46:22.089100 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:22.170368 8028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:22.170567 8028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:22.170588 8028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:22.171741 8028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:22.171780 8028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:22.171794 8028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:22.171808 8028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:22.171857 8028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:22.173348 8028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:22.173623 8028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:22.173843 8028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:22.178622 8033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:22.178644 8034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:22.178639 8036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:22.178675 8028 server_base.cc:1034] running on GCE node
I20260924 13:46:22.179064 8028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:22.179375 8028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:22.180541 8028 hybrid_clock.cc:648] HybridClock initialized: now 1790257582180536 us; error 34 us; skew 500 ppm
Sep 24 13:46:22 dist-test-slave-tfm9 krb5kdc[8015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:22.183081 8028 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:46:22.183547 8028 webserver.cc:460] Webserver started at http://127.0.0.1:39461/ using document root <none> and password file <none>
I20260924 13:46:22.183799 8028 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:22.183866 8028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:22.184005 8028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:22.185010 8028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "e375280dad68491ea31fdcba55f2ce36"
format_stamp: "Formatted at 2026-09-24 13:46:22 on dist-test-slave-tfm9"
server_key: "1669eb924e440b7050daedfa91e7107b"
server_key_iv: "3f27a3d75816d452b4a5d9c2fb3c163e"
server_key_version: "encryptionkey@0"
I20260924 13:46:22.185395 8028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "e375280dad68491ea31fdcba55f2ce36"
format_stamp: "Formatted at 2026-09-24 13:46:22 on dist-test-slave-tfm9"
server_key: "1669eb924e440b7050daedfa91e7107b"
server_key_iv: "3f27a3d75816d452b4a5d9c2fb3c163e"
server_key_version: "encryptionkey@0"
I20260924 13:46:22.186808 8028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:22.187553 8043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:22.187717 8028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:46:22.187840 8028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "e375280dad68491ea31fdcba55f2ce36"
format_stamp: "Formatted at 2026-09-24 13:46:22 on dist-test-slave-tfm9"
server_key: "1669eb924e440b7050daedfa91e7107b"
server_key_iv: "3f27a3d75816d452b4a5d9c2fb3c163e"
server_key_version: "encryptionkey@0"
I20260924 13:46:22.187916 8028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:22.191360 8028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:22.193147 8028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:22.193306 8028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:22.197234 8028 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38493
I20260924 13:46:22.197268 8095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38493 every 8 connection(s)
I20260924 13:46:22.197592 8028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:22.198480 8096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:22.200594 8096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36: Bootstrap starting.
I20260924 13:46:22.201082 8096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:22.201314 8096 log.cc:824] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:22.202003 8096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36: No bootstrap required, opened a new log
I20260924 13:46:22.203521 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 8028
I20260924 13:46:22.203665 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:22.203579 8096 raft_consensus.cc:348] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e375280dad68491ea31fdcba55f2ce36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38493 } }
I20260924 13:46:22.203789 8096 raft_consensus.cc:374] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:22.203860 8096 raft_consensus.cc:729] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e375280dad68491ea31fdcba55f2ce36, State: Initialized, Role: FOLLOWER
I20260924 13:46:22.203896 19671 external_mini_cluster.cc:1428] Setting key 3c43c1b8646e215a7af0c7d0bbcd3a51
I20260924 13:46:22.204013 8096 consensus_queue.cc:260] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e375280dad68491ea31fdcba55f2ce36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38493 } }
I20260924 13:46:22.204110 8096 raft_consensus.cc:388] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:22.204147 8096 raft_consensus.cc:482] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:22.204201 8096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:22.205410 8096 raft_consensus.cc:504] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e375280dad68491ea31fdcba55f2ce36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38493 } }
I20260924 13:46:22.205523 8096 leader_election.cc:302] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e375280dad68491ea31fdcba55f2ce36; no voters:
I20260924 13:46:22.205672 8096 leader_election.cc:288] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:22.205844 8102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:22.205960 8096 sys_catalog.cc:564] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:22.206050 8102 raft_consensus.cc:686] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [term 1 LEADER]: Becoming Leader. State: Replica: e375280dad68491ea31fdcba55f2ce36, State: Running, Role: LEADER
I20260924 13:46:22.206147 8102 consensus_queue.cc:237] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e375280dad68491ea31fdcba55f2ce36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38493 } }
I20260924 13:46:22.206287 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:22.204414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45315 (local address 127.0.0.1:38493)
0924 13:46:22.204640 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:22.204649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:22.204688 (+ 39us) server_negotiation.cc:407] Connection header received
0924 13:46:22.204834 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:22.204843 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:22.204894 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:22.205053 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:22.205691 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:22.205695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:22.205695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:22.205700 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:22.205748 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:22.205776 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:22.205851 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:22.205937 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 13:46:22.206106 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":99,"threads_started":1}
I20260924 13:46:22.206489 8102 sys_catalog.cc:455] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e375280dad68491ea31fdcba55f2ce36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e375280dad68491ea31fdcba55f2ce36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38493 } } }
I20260924 13:46:22.206614 8102 sys_catalog.cc:458] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:22.206787 8102 sys_catalog.cc:455] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e375280dad68491ea31fdcba55f2ce36. Latest consensus state: current_term: 1 leader_uuid: "e375280dad68491ea31fdcba55f2ce36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e375280dad68491ea31fdcba55f2ce36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38493 } } }
I20260924 13:46:22.206946 8102 sys_catalog.cc:458] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:22.207264 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:22.207348 8110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:22.207867 8110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:22.209384 8110 catalog_manager.cc:1347] Generated new cluster ID: 7a21e0c479834ed092ac9d354f849079
I20260924 13:46:22.209446 8110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:22.209933 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:22.208829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51189 (local address 127.0.0.1:38493)
0924 13:46:22.208864 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:22.208865 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:22.208990 (+ 125us) server_negotiation.cc:407] Connection header received
0924 13:46:22.209065 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:22.209066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:22.209071 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:22.209132 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:22.209692 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:22.209693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:22.209693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:22.209697 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:22.209711 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:22.209715 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:22.209776 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:22.209842 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 13:46:22.209857 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:46:22.219637 8110 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:22.219779 8110 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:22.223038 8110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e375280dad68491ea31fdcba55f2ce36: Generated new TSK 0
I20260924 13:46:22.223191 8110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:22.226604 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 8028
2026-09-24T13:46:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:22 dist-test-slave-tfm9 krb5kdc[8127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:22 dist-test-slave-tfm9 krb5kdc[8127](info): set up 2 sockets
Sep 24 13:46:22 dist-test-slave-tfm9 krb5kdc[8127](info): commencing operation
krb5kdc: starting...
W20260924 13:46:23.255797 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:23 dist-test-slave-tfm9 krb5kdc[8127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:46:23.372267 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35865
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:23.451754 8140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:23.451943 8140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:23.451964 8140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:23.452996 8140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:23.453040 8140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:23.453054 8140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:23.453068 8140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:23.453078 8140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:23.454416 8140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:23.454588 8140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:23.454741 8140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:23.459450 8148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:23.459441 8146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:23.459483 8140 server_base.cc:1034] running on GCE node
W20260924 13:46:23.459466 8145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:23.459903 8140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:23.460222 8140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:23.461403 8140 hybrid_clock.cc:648] HybridClock initialized: now 1790257583461400 us; error 35 us; skew 500 ppm
Sep 24 13:46:23 dist-test-slave-tfm9 krb5kdc[8127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:23.463919 8140 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:46:23.464406 8140 webserver.cc:460] Webserver started at http://127.0.0.1:43471/ using document root <none> and password file <none>
I20260924 13:46:23.464629 8140 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:23.464694 8140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:23.464818 8140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:23.465852 8140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "7ba0451a2d7f42a98844d23fe45e523f"
format_stamp: "Formatted at 2026-09-24 13:46:23 on dist-test-slave-tfm9"
server_key: "a00b9f47c145263fe80ef17774d56fc9"
server_key_iv: "cc0c7b27f5d2d522b770b9a9b2224815"
server_key_version: "encryptionkey@0"
I20260924 13:46:23.466281 8140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "7ba0451a2d7f42a98844d23fe45e523f"
format_stamp: "Formatted at 2026-09-24 13:46:23 on dist-test-slave-tfm9"
server_key: "a00b9f47c145263fe80ef17774d56fc9"
server_key_iv: "cc0c7b27f5d2d522b770b9a9b2224815"
server_key_version: "encryptionkey@0"
I20260924 13:46:23.467600 8140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:23.468313 8155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:23.468483 8140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:23.468581 8140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "7ba0451a2d7f42a98844d23fe45e523f"
format_stamp: "Formatted at 2026-09-24 13:46:23 on dist-test-slave-tfm9"
server_key: "a00b9f47c145263fe80ef17774d56fc9"
server_key_iv: "cc0c7b27f5d2d522b770b9a9b2224815"
server_key_version: "encryptionkey@0"
I20260924 13:46:23.468667 8140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:23.491284 8140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:23.493721 8140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:23.493885 8140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:23.498041 8140 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35865
I20260924 13:46:23.498080 8207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35865 every 8 connection(s)
I20260924 13:46:23.498440 8140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:23.499243 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:23.501428 8208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f: Bootstrap starting.
I20260924 13:46:23.502003 8208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:23.502290 8208 log.cc:824] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:23.502951 8208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f: No bootstrap required, opened a new log
I20260924 13:46:23.504544 8208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba0451a2d7f42a98844d23fe45e523f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35865 } }
I20260924 13:46:23.504675 8208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:23.504700 8208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba0451a2d7f42a98844d23fe45e523f, State: Initialized, Role: FOLLOWER
I20260924 13:46:23.504871 8208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba0451a2d7f42a98844d23fe45e523f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35865 } }
I20260924 13:46:23.504951 8208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:23.504992 8208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:23.505039 8208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:23.505543 8208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba0451a2d7f42a98844d23fe45e523f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35865 } }
I20260924 13:46:23.505650 8208 leader_election.cc:302] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ba0451a2d7f42a98844d23fe45e523f; no voters:
I20260924 13:46:23.505825 8208 leader_election.cc:288] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:23.505858 8211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:23.506042 8211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [term 1 LEADER]: Becoming Leader. State: Replica: 7ba0451a2d7f42a98844d23fe45e523f, State: Running, Role: LEADER
I20260924 13:46:23.506093 8208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:23.506161 8211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba0451a2d7f42a98844d23fe45e523f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35865 } }
I20260924 13:46:23.506625 8213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ba0451a2d7f42a98844d23fe45e523f. Latest consensus state: current_term: 1 leader_uuid: "7ba0451a2d7f42a98844d23fe45e523f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba0451a2d7f42a98844d23fe45e523f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35865 } } }
I20260924 13:46:23.506722 8213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:23.506753 8212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ba0451a2d7f42a98844d23fe45e523f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba0451a2d7f42a98844d23fe45e523f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35865 } } }
I20260924 13:46:23.506845 8212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:23.507254 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 8140
I20260924 13:46:23.507375 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:23.507588 19671 external_mini_cluster.cc:1428] Setting key 8a21b56deb6f0c15c224db5d5eff45e3
W20260924 13:46:23.507993 8227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:46:23.508088 8227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:46:23.508193 8220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:23.508360 8220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:23.510314 8220 catalog_manager.cc:1347] Generated new cluster ID: 5bd7989f65ed42e4a9b1ddcbd61bcf29
I20260924 13:46:23.510365 8220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:23.519411 8220 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:23.520048 8220 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 13:46:23 dist-test-slave-tfm9 krb5kdc[8127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 13:46:23.526398 8220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ba0451a2d7f42a98844d23fe45e523f: Generated new TSK 0
I20260924 13:46:23.526567 8220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:23.527642 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:23.508075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:46341 (local address 127.0.0.1:35865)
0924 13:46:23.509027 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:23.509035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:23.509061 (+ 26us) server_negotiation.cc:407] Connection header received
0924 13:46:23.509178 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:23.509185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:23.509245 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:23.509392 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:23.510089 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:23.519283 (+ 9194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:23.520334 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:23.520948 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:23.521062 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:23.523852 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:23.523856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:23.523856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:23.523868 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:23.525821 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:23.526389 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:23.526392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:23.526393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:23.526518 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:23.526784 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:23.526784 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:23.526785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:23.527027 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:23.527156 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:23.527312 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 13:46:23.527389 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":91,"threads_started":1}
I20260924 13:46:23.528548 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:23.536419 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:23.530034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:35711 (local address 127.0.0.1:35865)
0924 13:46:23.530074 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:23.530076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:23.530152 (+ 76us) server_negotiation.cc:407] Connection header received
0924 13:46:23.530219 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:23.530219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:23.530227 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:23.530276 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:23.530896 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:23.531394 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:23.532417 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:23.532598 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:23.532652 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:23.533970 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:23.533971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:23.533972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:23.533978 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:23.535344 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:23.535741 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:23.535742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:23.535742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:23.535788 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:23.536004 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:23.536005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:23.536005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:23.536091 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:23.536163 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:23.536308 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 13:46:23.536351 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:46:23.538511 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 8140
W20260924 13:46:23.542640 8125 connection.cc:535] client connection to 127.0.0.1:35865 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35865): Connection reset by peer (error 104)
2026-09-24T13:46:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:23 dist-test-slave-tfm9 krb5kdc[8239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:23 dist-test-slave-tfm9 krb5kdc[8239](info): set up 2 sockets
Sep 24 13:46:23 dist-test-slave-tfm9 krb5kdc[8239](info): commencing operation
krb5kdc: starting...
W20260924 13:46:24.567627 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:24 dist-test-slave-tfm9 krb5kdc[8239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:46:24.682451 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:24.761718 8252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:24.761935 8252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:24.761958 8252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:24.763018 8252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:24.763051 8252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:24.763065 8252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:24.763077 8252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:24.763089 8252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:24.764732 8252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:24.764921 8252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:24.765080 8252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:24.770018 8257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:24.770061 8260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:24.770218 8252 server_base.cc:1034] running on GCE node
W20260924 13:46:24.770249 8258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:24.770530 8252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:24.770874 8252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:24.772058 8252 hybrid_clock.cc:648] HybridClock initialized: now 1790257584772029 us; error 47 us; skew 500 ppm
Sep 24 13:46:24 dist-test-slave-tfm9 krb5kdc[8239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:24.774528 8252 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:46:24.774971 8252 webserver.cc:460] Webserver started at http://127.0.0.1:41705/ using document root <none> and password file <none>
I20260924 13:46:24.775195 8252 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:24.775266 8252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:24.775424 8252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:24.776542 8252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "480da81012fd4418957bfa32ddd8314a"
format_stamp: "Formatted at 2026-09-24 13:46:24 on dist-test-slave-tfm9"
server_key: "84d058ac22bbbb2ec3d11723c35153ea"
server_key_iv: "4413dd34f0245df92a3d5b7141077f5b"
server_key_version: "encryptionkey@0"
I20260924 13:46:24.776909 8252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "480da81012fd4418957bfa32ddd8314a"
format_stamp: "Formatted at 2026-09-24 13:46:24 on dist-test-slave-tfm9"
server_key: "84d058ac22bbbb2ec3d11723c35153ea"
server_key_iv: "4413dd34f0245df92a3d5b7141077f5b"
server_key_version: "encryptionkey@0"
I20260924 13:46:24.778288 8252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:24.779135 8267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:24.779351 8252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:46:24.779462 8252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "480da81012fd4418957bfa32ddd8314a"
format_stamp: "Formatted at 2026-09-24 13:46:24 on dist-test-slave-tfm9"
server_key: "84d058ac22bbbb2ec3d11723c35153ea"
server_key_iv: "4413dd34f0245df92a3d5b7141077f5b"
server_key_version: "encryptionkey@0"
I20260924 13:46:24.779538 8252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:24.792568 8252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:24.794571 8252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:24.794732 8252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:24.798954 8252 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36313
I20260924 13:46:24.799053 8319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36313 every 8 connection(s)
I20260924 13:46:24.799376 8252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:24.800274 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:24.802750 8320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a: Bootstrap starting.
I20260924 13:46:24.803364 8320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:24.803716 8320 log.cc:824] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:24.804422 8320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a: No bootstrap required, opened a new log
I20260924 13:46:24.806173 8320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480da81012fd4418957bfa32ddd8314a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } }
I20260924 13:46:24.806316 8320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:24.806366 8320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 480da81012fd4418957bfa32ddd8314a, State: Initialized, Role: FOLLOWER
I20260924 13:46:24.806545 8320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480da81012fd4418957bfa32ddd8314a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } }
I20260924 13:46:24.806612 8320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:24.806656 8320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:24.806721 8320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:24.806838 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 8252
I20260924 13:46:24.807003 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:24.807199 19671 external_mini_cluster.cc:1428] Setting key aefa728608919104e9fb3d09e97b79c0
I20260924 13:46:24.807557 8320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480da81012fd4418957bfa32ddd8314a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } }
I20260924 13:46:24.807669 8320 leader_election.cc:302] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 480da81012fd4418957bfa32ddd8314a; no voters:
I20260924 13:46:24.807849 8320 leader_election.cc:288] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:24.807987 8325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:24.808159 8325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [term 1 LEADER]: Becoming Leader. State: Replica: 480da81012fd4418957bfa32ddd8314a, State: Running, Role: LEADER
I20260924 13:46:24.808158 8320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:24.808291 8325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480da81012fd4418957bfa32ddd8314a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } }
I20260924 13:46:24.808700 8325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 480da81012fd4418957bfa32ddd8314a. Latest consensus state: current_term: 1 leader_uuid: "480da81012fd4418957bfa32ddd8314a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480da81012fd4418957bfa32ddd8314a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } } }
I20260924 13:46:24.809015 8325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:24.808892 8326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "480da81012fd4418957bfa32ddd8314a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480da81012fd4418957bfa32ddd8314a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } } }
I20260924 13:46:24.809484 8326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:24.809748 8334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:24.810403 8334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:24.811951 8334 catalog_manager.cc:1347] Generated new cluster ID: d57c4851ac39481d891dbb5a2e075835
I20260924 13:46:24.811995 8334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:24.814015 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:24.807699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:33819 (local address 127.0.0.1:36313)
0924 13:46:24.807868 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:24.807877 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:24.807938 (+ 61us) server_negotiation.cc:407] Connection header received
0924 13:46:24.808061 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:24.808067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:24.808124 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:24.808491 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:24.808561 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:24.810411 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:24.811665 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:24.812275 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:24.812387 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:24.813169 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:24.813173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:24.813173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:24.813380 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:24.813416 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:24.813429 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:24.813509 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:24.813643 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 13:46:24.813767 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260924 13:46:24.815109 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:24.820459 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:24.816746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:52521 (local address 127.0.0.1:36313)
0924 13:46:24.816787 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:24.816789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:24.816899 (+ 110us) server_negotiation.cc:407] Connection header received
0924 13:46:24.816926 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:24.816927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:24.816938 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:24.817014 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:24.817145 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:24.818019 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:24.819026 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:24.819386 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:24.819466 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:24.820175 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:24.820176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:24.820176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:24.820181 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:24.820197 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:24.820211 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:24.820297 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:24.820369 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 13:46:24.820382 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:46:24.825011 8334 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:24.825532 8334 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:24.829077 8334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 480da81012fd4418957bfa32ddd8314a: Generated new TSK 0
I20260924 13:46:24.829242 8334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:24.836452 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 8252
W20260924 13:46:24.840623 8237 connection.cc:535] client connection to 127.0.0.1:36313 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36313): Connection reset by peer (error 104)
2026-09-24T13:46:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:24 dist-test-slave-tfm9 krb5kdc[8351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:24 dist-test-slave-tfm9 krb5kdc[8351](info): set up 2 sockets
Sep 24 13:46:24 dist-test-slave-tfm9 krb5kdc[8351](info): commencing operation
krb5kdc: starting...
W20260924 13:46:25.904323 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:25 dist-test-slave-tfm9 krb5kdc[8351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:46:26.024674 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:26.105993 8364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:26.106187 8364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:26.106209 8364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:26.107290 8364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:26.107324 8364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:26.107339 8364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:26.107352 8364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:26.107364 8364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:26.108735 8364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:26.108997 8364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:26.109175 8364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:26.113898 8369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:26.113905 8370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:26.113930 8372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:26.114162 8364 server_base.cc:1034] running on GCE node
I20260924 13:46:26.114441 8364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:26.114760 8364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:26.115911 8364 hybrid_clock.cc:648] HybridClock initialized: now 1790257586115890 us; error 42 us; skew 500 ppm
Sep 24 13:46:26 dist-test-slave-tfm9 krb5kdc[8351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:26.118397 8364 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:46:26.118857 8364 webserver.cc:460] Webserver started at http://127.0.0.1:32815/ using document root <none> and password file <none>
I20260924 13:46:26.119140 8364 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:26.119205 8364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:26.119344 8364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:26.120368 8364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "11a2cd59dd6041be90a73326bcad655e"
format_stamp: "Formatted at 2026-09-24 13:46:26 on dist-test-slave-tfm9"
server_key: "6d95c0361eeba3548f864883befcf16a"
server_key_iv: "eecf42a37aad915a202aa354e7bfd45e"
server_key_version: "encryptionkey@0"
I20260924 13:46:26.120726 8364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "11a2cd59dd6041be90a73326bcad655e"
format_stamp: "Formatted at 2026-09-24 13:46:26 on dist-test-slave-tfm9"
server_key: "6d95c0361eeba3548f864883befcf16a"
server_key_iv: "eecf42a37aad915a202aa354e7bfd45e"
server_key_version: "encryptionkey@0"
I20260924 13:46:26.122037 8364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:46:26.122737 8379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:26.122908 8364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:26.123015 8364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "11a2cd59dd6041be90a73326bcad655e"
format_stamp: "Formatted at 2026-09-24 13:46:26 on dist-test-slave-tfm9"
server_key: "6d95c0361eeba3548f864883befcf16a"
server_key_iv: "eecf42a37aad915a202aa354e7bfd45e"
server_key_version: "encryptionkey@0"
I20260924 13:46:26.123100 8364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:26.126283 8364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:26.128017 8364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:26.128166 8364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:26.132179 8364 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45457
I20260924 13:46:26.132217 8431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45457 every 8 connection(s)
I20260924 13:46:26.132534 8364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:26.133419 8432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:26.135744 8432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e: Bootstrap starting.
I20260924 13:46:26.136245 8432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:26.136482 8432 log.cc:824] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:26.137099 8432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e: No bootstrap required, opened a new log
I20260924 13:46:26.138754 8432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a2cd59dd6041be90a73326bcad655e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260924 13:46:26.138887 8432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:26.138914 8432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a2cd59dd6041be90a73326bcad655e, State: Initialized, Role: FOLLOWER
I20260924 13:46:26.139055 8432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a2cd59dd6041be90a73326bcad655e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260924 13:46:26.139134 8432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:26.139156 8432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:26.139185 8432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:26.139418 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 8364
I20260924 13:46:26.139518 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:26.139698 19671 external_mini_cluster.cc:1428] Setting key 47bfea1c34c1897ea5ac62a994d6db40
I20260924 13:46:26.139786 8432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a2cd59dd6041be90a73326bcad655e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260924 13:46:26.139927 8432 leader_election.cc:302] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11a2cd59dd6041be90a73326bcad655e; no voters:
I20260924 13:46:26.140089 8432 leader_election.cc:288] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:26.140141 8435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:26.140298 8435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [term 1 LEADER]: Becoming Leader. State: Replica: 11a2cd59dd6041be90a73326bcad655e, State: Running, Role: LEADER
I20260924 13:46:26.140336 8432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:26.140367 8435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a2cd59dd6041be90a73326bcad655e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260924 13:46:26.140666 8435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11a2cd59dd6041be90a73326bcad655e. Latest consensus state: current_term: 1 leader_uuid: "11a2cd59dd6041be90a73326bcad655e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a2cd59dd6041be90a73326bcad655e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } } }
I20260924 13:46:26.140676 8436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11a2cd59dd6041be90a73326bcad655e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a2cd59dd6041be90a73326bcad655e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } } }
I20260924 13:46:26.140841 8435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:26.140875 8436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:26.141676 8441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:26.142328 8441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:26.142565 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:26.140305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:56617 (local address 127.0.0.1:45457)
0924 13:46:26.140522 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:26.140531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:26.140776 (+ 245us) server_negotiation.cc:407] Connection header received
0924 13:46:26.141077 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:26.141084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:26.141141 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:26.141592 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:26.142054 (+ 462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:26.142057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:26.142058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:26.142061 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:26.142106 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:26.142121 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:26.142191 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:26.142336 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 13:46:26.142381 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
I20260924 13:46:26.143815 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:26.143968 8441 catalog_manager.cc:1347] Generated new cluster ID: 6fc8a58d716f49d6b5b6e1c61f25a890
I20260924 13:46:26.144022 8441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:26.146241 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:26.145116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:51411 (local address 127.0.0.1:45457)
0924 13:46:26.145151 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:26.145152 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:26.145298 (+ 146us) server_negotiation.cc:407] Connection header received
0924 13:46:26.145345 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:26.145345 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:26.145351 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:26.145406 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:26.146012 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:26.146013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:26.146013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:26.146017 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:26.146031 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:26.146036 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:26.146093 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:26.146150 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 13:46:26.146162 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 13:46:26.166229 8441 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:26.166429 8441 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:26.183239 8441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11a2cd59dd6041be90a73326bcad655e: Generated new TSK 0
I20260924 13:46:26.183485 8441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:26.189342 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 8364
2026-09-24T13:46:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:26 dist-test-slave-tfm9 krb5kdc[8463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:26 dist-test-slave-tfm9 krb5kdc[8463](info): set up 2 sockets
Sep 24 13:46:26 dist-test-slave-tfm9 krb5kdc[8463](info): commencing operation
krb5kdc: starting...
W20260924 13:46:27.219801 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:27 dist-test-slave-tfm9 krb5kdc[8463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:46:27.336274 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:27.417872 8476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:27.418105 8476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:27.418130 8476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:27.419281 8476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:27.419325 8476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:27.419342 8476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:27.419356 8476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:27.419370 8476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:27.420790 8476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46045
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:27.420979 8476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:27.421198 8476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:46:27.426108 8476 server_base.cc:1034] running on GCE node
W20260924 13:46:27.426048 8484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:27.426075 8481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:27.426060 8482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:27.426539 8476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:27.426891 8476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:27.428081 8476 hybrid_clock.cc:648] HybridClock initialized: now 1790257587428068 us; error 42 us; skew 500 ppm
Sep 24 13:46:27 dist-test-slave-tfm9 krb5kdc[8463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:27.436810 8476 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:46:27.437316 8476 webserver.cc:460] Webserver started at http://127.0.0.1:37415/ using document root <none> and password file <none>
I20260924 13:46:27.437521 8476 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:27.437563 8476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:27.437651 8476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:27.438755 8476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "ace0e7939c5545cabf4525a82e87a640"
format_stamp: "Formatted at 2026-09-24 13:46:27 on dist-test-slave-tfm9"
server_key: "c74f0d2e0a603a8177fa25c0bef7e798"
server_key_iv: "b2f6e661c12171073c68cc91e5cd879e"
server_key_version: "encryptionkey@0"
I20260924 13:46:27.439059 8476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "ace0e7939c5545cabf4525a82e87a640"
format_stamp: "Formatted at 2026-09-24 13:46:27 on dist-test-slave-tfm9"
server_key: "c74f0d2e0a603a8177fa25c0bef7e798"
server_key_iv: "b2f6e661c12171073c68cc91e5cd879e"
server_key_version: "encryptionkey@0"
I20260924 13:46:27.440330 8476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:46:27.441030 8491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:27.441221 8476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:46:27.441298 8476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "ace0e7939c5545cabf4525a82e87a640"
format_stamp: "Formatted at 2026-09-24 13:46:27 on dist-test-slave-tfm9"
server_key: "c74f0d2e0a603a8177fa25c0bef7e798"
server_key_iv: "b2f6e661c12171073c68cc91e5cd879e"
server_key_version: "encryptionkey@0"
I20260924 13:46:27.441351 8476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:27.462224 8476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:27.464576 8476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:27.464689 8476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:27.468796 8476 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46045
I20260924 13:46:27.468871 8543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46045 every 8 connection(s)
I20260924 13:46:27.469205 8476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:27.470098 8544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:27.470698 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 8476
I20260924 13:46:27.470793 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:27.470973 19671 external_mini_cluster.cc:1428] Setting key ed652704204a10ab5dd00fea94ddcdb2
I20260924 13:46:27.473016 8544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640: Bootstrap starting.
I20260924 13:46:27.473609 8544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:27.473943 8544 log.cc:824] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:27.474726 8544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640: No bootstrap required, opened a new log
I20260924 13:46:27.477239 8544 raft_consensus.cc:348] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace0e7939c5545cabf4525a82e87a640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260924 13:46:27.477421 8544 raft_consensus.cc:374] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:27.477458 8544 raft_consensus.cc:729] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace0e7939c5545cabf4525a82e87a640, State: Initialized, Role: FOLLOWER
I20260924 13:46:27.477599 8544 consensus_queue.cc:260] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace0e7939c5545cabf4525a82e87a640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260924 13:46:27.477692 8544 raft_consensus.cc:388] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:27.477736 8544 raft_consensus.cc:482] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:27.477787 8544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 13:46:27 dist-test-slave-tfm9 krb5kdc[8463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 13:46:27.478408 8544 raft_consensus.cc:504] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace0e7939c5545cabf4525a82e87a640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260924 13:46:27.478520 8544 leader_election.cc:302] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ace0e7939c5545cabf4525a82e87a640; no voters:
I20260924 13:46:27.478673 8544 leader_election.cc:288] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:27.478699 8549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:27.478819 8549 raft_consensus.cc:686] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [term 1 LEADER]: Becoming Leader. State: Replica: ace0e7939c5545cabf4525a82e87a640, State: Running, Role: LEADER
I20260924 13:46:27.478938 8544 sys_catalog.cc:564] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:27.478989 8549 consensus_queue.cc:237] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace0e7939c5545cabf4525a82e87a640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260924 13:46:27.479301 8549 sys_catalog.cc:455] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ace0e7939c5545cabf4525a82e87a640. Latest consensus state: current_term: 1 leader_uuid: "ace0e7939c5545cabf4525a82e87a640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace0e7939c5545cabf4525a82e87a640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } } }
I20260924 13:46:27.479373 8549 sys_catalog.cc:458] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:27.479338 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ace0e7939c5545cabf4525a82e87a640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace0e7939c5545cabf4525a82e87a640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } } }
I20260924 13:46:27.479497 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640 [sys.catalog]: This master's current role is: LEADER
W20260924 13:46:27.480825 8564 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:46:27.480923 8564 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:46:27.481027 8554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:27.481176 8554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:27.482656 8554 catalog_manager.cc:1347] Generated new cluster ID: 0d4f493fa5c04fda88e2570107d3e579
I20260924 13:46:27.482729 8554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:27.483006 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:27.471480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:55737 (local address 127.0.0.1:46045)
0924 13:46:27.471630 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:27.471638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:27.471662 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:46:27.472079 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:27.472085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:27.472131 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:27.472243 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:27.473096 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:27.474303 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:27.475428 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:27.475858 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:27.475942 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:27.478724 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:27.478728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:27.478728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:27.478741 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:27.480864 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:27.481328 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:27.481331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:27.481331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:27.481433 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:27.481720 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:27.481721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:27.481722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:27.482068 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:27.482246 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:27.482570 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 13:46:27.482805 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260924 13:46:27.483952 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:27.493042 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:27.485414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:43943 (local address 127.0.0.1:46045)
0924 13:46:27.485449 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:27.485451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:27.485514 (+ 63us) server_negotiation.cc:407] Connection header received
0924 13:46:27.485565 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:27.485566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:27.485575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:27.485641 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:27.486222 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:27.487363 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:27.488371 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:27.488687 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:27.488763 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:27.490058 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:27.490059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:27.490059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:27.490066 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:27.491739 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:27.492184 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:27.492185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:27.492185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:27.492244 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:27.492511 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:27.492512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:27.492512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:27.492627 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:27.492732 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:27.492894 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 13:46:27.492956 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:46:27.508869 8554 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:27.509384 8554 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:27.517390 8554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ace0e7939c5545cabf4525a82e87a640: Generated new TSK 0
I20260924 13:46:27.517583 8554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:27.523370 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 8476
W20260924 13:46:27.527868 8461 connection.cc:535] client connection to 127.0.0.1:46045 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46045): Connection reset by peer (error 104)
2026-09-24T13:46:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:27 dist-test-slave-tfm9 krb5kdc[8575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:27 dist-test-slave-tfm9 krb5kdc[8575](info): set up 2 sockets
Sep 24 13:46:27 dist-test-slave-tfm9 krb5kdc[8575](info): commencing operation
krb5kdc: starting...
W20260924 13:46:28.563566 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:28 dist-test-slave-tfm9 krb5kdc[8575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:46:28.677322 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:28.756983 8588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:28.757225 8588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:28.757264 8588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:28.758440 8588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:28.758507 8588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:28.758531 8588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:28.758564 8588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:28.758589 8588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:28.760025 8588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:28.760295 8588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:28.760515 8588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:28.765394 8596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:28.765457 8588 server_base.cc:1034] running on GCE node
W20260924 13:46:28.765401 8594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:28.765403 8593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:28.765818 8588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:28.766142 8588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:28.767287 8588 hybrid_clock.cc:648] HybridClock initialized: now 1790257588767279 us; error 32 us; skew 500 ppm
Sep 24 13:46:28 dist-test-slave-tfm9 krb5kdc[8575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:28.769645 8588 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:46:28.770129 8588 webserver.cc:460] Webserver started at http://127.0.0.1:39435/ using document root <none> and password file <none>
I20260924 13:46:28.770351 8588 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:28.770418 8588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:28.770562 8588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:28.771581 8588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "c836e7fa0fbb471a9ec9a344ed2b464e"
format_stamp: "Formatted at 2026-09-24 13:46:28 on dist-test-slave-tfm9"
server_key: "13bddf497b49bf26d52369b7e392c3e1"
server_key_iv: "319ab5879a51821a13d1f97dda59dcc4"
server_key_version: "encryptionkey@0"
I20260924 13:46:28.771941 8588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "c836e7fa0fbb471a9ec9a344ed2b464e"
format_stamp: "Formatted at 2026-09-24 13:46:28 on dist-test-slave-tfm9"
server_key: "13bddf497b49bf26d52369b7e392c3e1"
server_key_iv: "319ab5879a51821a13d1f97dda59dcc4"
server_key_version: "encryptionkey@0"
I20260924 13:46:28.773222 8588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:46:28.773977 8603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:28.774152 8588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:28.774246 8588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "c836e7fa0fbb471a9ec9a344ed2b464e"
format_stamp: "Formatted at 2026-09-24 13:46:28 on dist-test-slave-tfm9"
server_key: "13bddf497b49bf26d52369b7e392c3e1"
server_key_iv: "319ab5879a51821a13d1f97dda59dcc4"
server_key_version: "encryptionkey@0"
I20260924 13:46:28.774328 8588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:28.789507 8588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:28.791291 8588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:28.791457 8588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:28.795204 8588 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42037
I20260924 13:46:28.795230 8655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42037 every 8 connection(s)
I20260924 13:46:28.795591 8588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:28.796396 8656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:28.798729 8656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e: Bootstrap starting.
I20260924 13:46:28.799226 8656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:28.799449 8656 log.cc:824] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:28.800047 8656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e: No bootstrap required, opened a new log
I20260924 13:46:28.801447 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 8588
I20260924 13:46:28.801561 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:28.801512 8656 raft_consensus.cc:348] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836e7fa0fbb471a9ec9a344ed2b464e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260924 13:46:28.801705 8656 raft_consensus.cc:374] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:28.801753 19671 external_mini_cluster.cc:1428] Setting key 3997f5635163950cff09439dc9b8e9cb
I20260924 13:46:28.801772 8656 raft_consensus.cc:729] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c836e7fa0fbb471a9ec9a344ed2b464e, State: Initialized, Role: FOLLOWER
I20260924 13:46:28.801928 8656 consensus_queue.cc:260] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836e7fa0fbb471a9ec9a344ed2b464e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260924 13:46:28.802016 8656 raft_consensus.cc:388] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:28.802058 8656 raft_consensus.cc:482] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:28.802107 8656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:28.803092 8656 raft_consensus.cc:504] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836e7fa0fbb471a9ec9a344ed2b464e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260924 13:46:28.803220 8656 leader_election.cc:302] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c836e7fa0fbb471a9ec9a344ed2b464e; no voters:
I20260924 13:46:28.803378 8656 leader_election.cc:288] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:28.803421 8661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:28.803587 8661 raft_consensus.cc:686] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [term 1 LEADER]: Becoming Leader. State: Replica: c836e7fa0fbb471a9ec9a344ed2b464e, State: Running, Role: LEADER
I20260924 13:46:28.803620 8656 sys_catalog.cc:564] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:28.803730 8661 consensus_queue.cc:237] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836e7fa0fbb471a9ec9a344ed2b464e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260924 13:46:28.804143 8662 sys_catalog.cc:455] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c836e7fa0fbb471a9ec9a344ed2b464e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836e7fa0fbb471a9ec9a344ed2b464e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } } }
I20260924 13:46:28.804205 8662 sys_catalog.cc:458] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:28.804414 8663 sys_catalog.cc:455] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c836e7fa0fbb471a9ec9a344ed2b464e. Latest consensus state: current_term: 1 leader_uuid: "c836e7fa0fbb471a9ec9a344ed2b464e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836e7fa0fbb471a9ec9a344ed2b464e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } } }
I20260924 13:46:28.804528 8663 sys_catalog.cc:458] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:28.804687 8668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:28.805166 8668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:28.806831 8668 catalog_manager.cc:1347] Generated new cluster ID: 71b4ece7a41c4a75b88532c3b5b106ca
I20260924 13:46:28.806913 8668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:28.807740 8660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:28.802375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:45395 (local address 127.0.0.1:42037)
0924 13:46:28.802541 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:28.802549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:28.802582 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:46:28.802677 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:28.802681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:28.802722 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:28.802836 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:28.803071 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:28.804353 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:28.805472 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:28.806176 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:28.806252 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:28.806980 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:28.806983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:28.806984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:28.807178 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:28.807217 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:28.807231 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:28.807322 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:28.807507 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 13:46:28.807536 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260924 13:46:28.808741 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:28.813539 8660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:28.810109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:57519 (local address 127.0.0.1:42037)
0924 13:46:28.810143 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:28.810145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:28.810297 (+ 152us) server_negotiation.cc:407] Connection header received
0924 13:46:28.810335 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:28.810336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:28.810347 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:28.810403 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:28.810552 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:28.811263 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:28.812222 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:28.812539 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:28.812614 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:28.813283 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:28.813285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:28.813285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:28.813289 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:28.813304 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:28.813308 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:28.813373 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:28.813442 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 13:46:28.813466 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:46:28.820011 8668 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:28.820518 8668 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:28.823637 8668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c836e7fa0fbb471a9ec9a344ed2b464e: Generated new TSK 0
I20260924 13:46:28.823828 8668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:28.830276 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 8588
W20260924 13:46:28.834470 8573 connection.cc:535] client connection to 127.0.0.1:42037 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42037): Connection reset by peer (error 104)
2026-09-24T13:46:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:28 dist-test-slave-tfm9 krb5kdc[8687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:28 dist-test-slave-tfm9 krb5kdc[8687](info): set up 2 sockets
Sep 24 13:46:28 dist-test-slave-tfm9 krb5kdc[8687](info): commencing operation
krb5kdc: starting...
W20260924 13:46:29.863528 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:29 dist-test-slave-tfm9 krb5kdc[8687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:46:29.982488 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:30.062753 8700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:30.062951 8700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:30.062974 8700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:30.064088 8700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:30.064144 8700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:30.064158 8700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:30.064170 8700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:30.064182 8700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:30.065661 8700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:30.065918 8700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:30.066202 8700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:46:30.071024 8700 server_base.cc:1034] running on GCE node
W20260924 13:46:30.071066 8706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:30.070997 8708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:30.070993 8705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:30.071388 8700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:30.071717 8700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:30.072904 8700 hybrid_clock.cc:648] HybridClock initialized: now 1790257590072877 us; error 40 us; skew 500 ppm
Sep 24 13:46:30 dist-test-slave-tfm9 krb5kdc[8687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:30.075305 8700 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:46:30.075763 8700 webserver.cc:460] Webserver started at http://127.0.0.1:35585/ using document root <none> and password file <none>
I20260924 13:46:30.075982 8700 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:30.076049 8700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:30.076195 8700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:30.077231 8700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "a9eb0c883947429f90ace67e0b0bbb86"
format_stamp: "Formatted at 2026-09-24 13:46:30 on dist-test-slave-tfm9"
server_key: "93367d0ad09a9d5d5d83e23c7b7ace51"
server_key_iv: "3c4058b13b67d2618fc59440c6632449"
server_key_version: "encryptionkey@0"
I20260924 13:46:30.077582 8700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "a9eb0c883947429f90ace67e0b0bbb86"
format_stamp: "Formatted at 2026-09-24 13:46:30 on dist-test-slave-tfm9"
server_key: "93367d0ad09a9d5d5d83e23c7b7ace51"
server_key_iv: "3c4058b13b67d2618fc59440c6632449"
server_key_version: "encryptionkey@0"
I20260924 13:46:30.078946 8700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:30.079708 8715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:30.079878 8700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:30.079975 8700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "a9eb0c883947429f90ace67e0b0bbb86"
format_stamp: "Formatted at 2026-09-24 13:46:30 on dist-test-slave-tfm9"
server_key: "93367d0ad09a9d5d5d83e23c7b7ace51"
server_key_iv: "3c4058b13b67d2618fc59440c6632449"
server_key_version: "encryptionkey@0"
I20260924 13:46:30.080044 8700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:30.083365 8700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:30.085130 8700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:30.085294 8700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:30.089365 8700 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35071
I20260924 13:46:30.089397 8767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35071 every 8 connection(s)
I20260924 13:46:30.089752 8700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:30.090631 8768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:30.093047 8768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86: Bootstrap starting.
I20260924 13:46:30.093575 8768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:30.093849 8768 log.cc:824] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:30.094580 8768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86: No bootstrap required, opened a new log
I20260924 13:46:30.096278 8768 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9eb0c883947429f90ace67e0b0bbb86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } }
I20260924 13:46:30.096431 8768 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:30.096457 8768 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9eb0c883947429f90ace67e0b0bbb86, State: Initialized, Role: FOLLOWER
I20260924 13:46:30.096563 8768 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9eb0c883947429f90ace67e0b0bbb86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } }
I20260924 13:46:30.096619 8768 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:30.096642 8768 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:30.096720 8768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:30.097000 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 8700
I20260924 13:46:30.097081 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:30.097291 19671 external_mini_cluster.cc:1428] Setting key b91c5720fab0b77777a9c8165150e47b
I20260924 13:46:30.097445 8768 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9eb0c883947429f90ace67e0b0bbb86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } }
I20260924 13:46:30.097563 8768 leader_election.cc:302] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9eb0c883947429f90ace67e0b0bbb86; no voters:
I20260924 13:46:30.097731 8768 leader_election.cc:288] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:30.097783 8771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:30.097961 8771 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [term 1 LEADER]: Becoming Leader. State: Replica: a9eb0c883947429f90ace67e0b0bbb86, State: Running, Role: LEADER
I20260924 13:46:30.098052 8768 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:30.098059 8771 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9eb0c883947429f90ace67e0b0bbb86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } }
I20260924 13:46:30.098402 8771 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9eb0c883947429f90ace67e0b0bbb86. Latest consensus state: current_term: 1 leader_uuid: "a9eb0c883947429f90ace67e0b0bbb86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9eb0c883947429f90ace67e0b0bbb86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } } }
I20260924 13:46:30.098477 8771 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:30.098547 8773 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9eb0c883947429f90ace67e0b0bbb86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9eb0c883947429f90ace67e0b0bbb86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } } }
I20260924 13:46:30.098601 8773 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:30.098666 8778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:30.099128 8778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:30.100867 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:30.097777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:59869 (local address 127.0.0.1:35071)
0924 13:46:30.098151 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:30.098164 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:30.098389 (+ 225us) server_negotiation.cc:407] Connection header received
0924 13:46:30.098626 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:30.098636 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:30.098677 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:30.098807 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:30.100320 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:30.100323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:30.100323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:30.100347 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:30.100392 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:30.100409 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:30.100500 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:30.100577 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 13:46:30.100635 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":71,"threads_started":1}
I20260924 13:46:30.100939 8778 catalog_manager.cc:1347] Generated new cluster ID: 850e24f1db544bee8f937dc466560784
I20260924 13:46:30.100980 8778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:30.101768 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:30.107404 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:30.103233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:52077 (local address 127.0.0.1:35071)
0924 13:46:30.103281 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:30.103282 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:30.106312 (+ 3030us) server_negotiation.cc:407] Connection header received
0924 13:46:30.106360 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:30.106362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:30.106378 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:30.106429 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:30.107121 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:30.107122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:30.107122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:30.107127 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:30.107143 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:30.107148 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:30.107213 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:30.107284 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 13:46:30.107298 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:46:30.133473 8778 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:30.133677 8778 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:30.138936 8778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9eb0c883947429f90ace67e0b0bbb86: Generated new TSK 0
I20260924 13:46:30.139137 8778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:30.146646 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 8700
2026-09-24T13:46:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:30 dist-test-slave-tfm9 krb5kdc[8799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:30 dist-test-slave-tfm9 krb5kdc[8799](info): set up 2 sockets
Sep 24 13:46:30 dist-test-slave-tfm9 krb5kdc[8799](info): commencing operation
krb5kdc: starting...
W20260924 13:46:31.175802 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:31 dist-test-slave-tfm9 krb5kdc[8799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:46:31.292800 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:43275
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:43275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:31.372121 8812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:31.372360 8812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:31.372400 8812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:31.373539 8812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:31.373605 8812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:31.373628 8812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:31.373656 8812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:31.373680 8812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:31.375154 8812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:43275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:43275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:31.375428 8812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:31.375643 8812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:31.380541 8818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:31.380555 8817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:31.380582 8812 server_base.cc:1034] running on GCE node
W20260924 13:46:31.380553 8820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:31.381026 8812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:31.381340 8812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:31.382505 8812 hybrid_clock.cc:648] HybridClock initialized: now 1790257591382498 us; error 35 us; skew 500 ppm
Sep 24 13:46:31 dist-test-slave-tfm9 krb5kdc[8799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257591, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:31.385033 8812 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:46:31.385496 8812 webserver.cc:460] Webserver started at http://127.19.53.254:36507/ using document root <none> and password file <none>
I20260924 13:46:31.385726 8812 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:31.385802 8812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:31.385977 8812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:31.387100 8812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "29ddffbfd1094c7da1066c4d4799cb2a"
format_stamp: "Formatted at 2026-09-24 13:46:31 on dist-test-slave-tfm9"
server_key: "5079039ee4dd645dc4178b1bbcdf46b3"
server_key_iv: "32f1b91a5f62711eba1f3936be4b4c79"
server_key_version: "encryptionkey@0"
I20260924 13:46:31.387485 8812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "29ddffbfd1094c7da1066c4d4799cb2a"
format_stamp: "Formatted at 2026-09-24 13:46:31 on dist-test-slave-tfm9"
server_key: "5079039ee4dd645dc4178b1bbcdf46b3"
server_key_iv: "32f1b91a5f62711eba1f3936be4b4c79"
server_key_version: "encryptionkey@0"
I20260924 13:46:31.388827 8812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:46:31.389679 8827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:31.389878 8812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:31.390015 8812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "29ddffbfd1094c7da1066c4d4799cb2a"
format_stamp: "Formatted at 2026-09-24 13:46:31 on dist-test-slave-tfm9"
server_key: "5079039ee4dd645dc4178b1bbcdf46b3"
server_key_iv: "32f1b91a5f62711eba1f3936be4b4c79"
server_key_version: "encryptionkey@0"
I20260924 13:46:31.390105 8812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:31.404631 8812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:31.406644 8812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:31.406804 8812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:31.410862 8812 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:43275
I20260924 13:46:31.410894 8879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:43275 every 8 connection(s)
I20260924 13:46:31.411260 8812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:31.412127 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:31.414587 8880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a: Bootstrap starting.
I20260924 13:46:31.415150 8880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:31.415438 8880 log.cc:824] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:31.416105 8880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a: No bootstrap required, opened a new log
I20260924 13:46:31.417160 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 8812
I20260924 13:46:31.417274 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:31.417483 19671 external_mini_cluster.cc:1428] Setting key 7a5329b4cef74e77ee3da13196f56c99
I20260924 13:46:31.417804 8880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ddffbfd1094c7da1066c4d4799cb2a" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43275 } }
I20260924 13:46:31.418037 8880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:31.418100 8880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ddffbfd1094c7da1066c4d4799cb2a, State: Initialized, Role: FOLLOWER
I20260924 13:46:31.418290 8880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ddffbfd1094c7da1066c4d4799cb2a" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43275 } }
I20260924 13:46:31.418375 8880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:31.418397 8880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:31.418458 8880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:31.419569 8880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ddffbfd1094c7da1066c4d4799cb2a" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43275 } }
I20260924 13:46:31.419656 8880 leader_election.cc:302] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29ddffbfd1094c7da1066c4d4799cb2a; no voters:
I20260924 13:46:31.419858 8880 leader_election.cc:288] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:31.419878 8885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:31.419987 8885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [term 1 LEADER]: Becoming Leader. State: Replica: 29ddffbfd1094c7da1066c4d4799cb2a, State: Running, Role: LEADER
I20260924 13:46:31.420116 8880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:31.420120 8885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ddffbfd1094c7da1066c4d4799cb2a" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43275 } }
I20260924 13:46:31.420725 8886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29ddffbfd1094c7da1066c4d4799cb2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ddffbfd1094c7da1066c4d4799cb2a" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43275 } } }
I20260924 13:46:31.420740 8887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29ddffbfd1094c7da1066c4d4799cb2a. Latest consensus state: current_term: 1 leader_uuid: "29ddffbfd1094c7da1066c4d4799cb2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ddffbfd1094c7da1066c4d4799cb2a" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43275 } } }
I20260924 13:46:31.420852 8886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:31.420858 8887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:31.421471 8893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:31.422118 8893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:31.424557 8893 catalog_manager.cc:1347] Generated new cluster ID: 4c1a56bd87e14ad583b86c2827ef2af1
I20260924 13:46:31.424624 8893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:46:31 dist-test-slave-tfm9 krb5kdc[8799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:31.429934 8884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:31.418025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44527 (local address 127.19.53.254:43275)
0924 13:46:31.418200 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:31.418210 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:31.418239 (+ 29us) server_negotiation.cc:407] Connection header received
0924 13:46:31.418340 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:31.418346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:31.418413 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:31.418553 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:31.419363 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:31.421290 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:31.422357 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:31.422998 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:31.423103 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:31.425733 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:31.425738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:31.425738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:31.425750 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:31.427757 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:31.428239 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:31.428242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:31.428243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:31.428353 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:31.428605 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:31.428606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:31.428607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:31.428914 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:31.429515 (+ 601us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:31.429613 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 13:46:31.429710 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260924 13:46:31.431008 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:31.439396 8884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:31.432412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35355 (local address 127.19.53.254:43275)
0924 13:46:31.432445 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:31.432447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:31.432484 (+ 37us) server_negotiation.cc:407] Connection header received
0924 13:46:31.432539 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:31.432539 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:31.432548 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:31.432612 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:31.433179 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:31.433957 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:31.434912 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:31.435259 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:31.435327 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:31.436648 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:31.436649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:31.436649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:31.436656 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:31.438286 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:31.438656 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:31.438657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:31.438657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:31.438715 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:31.438927 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:31.438927 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:31.438928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:31.439050 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:31.439141 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:31.439259 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 13:46:31.439317 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:46:31.470263 8893 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:31.470880 8893 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:31.477099 8893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29ddffbfd1094c7da1066c4d4799cb2a: Generated new TSK 0
I20260924 13:46:31.477319 8893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:31.485441 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 8812
W20260924 13:46:31.490031 8797 connection.cc:535] client connection to 127.19.53.254:43275 recv error: Network error: failed to read from TLS socket (remote: 127.19.53.254:43275): Connection reset by peer (error 104)
2026-09-24T13:46:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:31 dist-test-slave-tfm9 krb5kdc[8911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:31 dist-test-slave-tfm9 krb5kdc[8911](info): set up 2 sockets
Sep 24 13:46:31 dist-test-slave-tfm9 krb5kdc[8911](info): commencing operation
krb5kdc: starting...
W20260924 13:46:32.515633 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:32 dist-test-slave-tfm9 krb5kdc[8911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:46:32.630903 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33481
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:41417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:33481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:32.710031 8924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:32.710235 8924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:32.710259 8924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:32.711400 8924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:32.711442 8924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:32.711457 8924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:32.711468 8924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:32.711480 8924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:32.712929 8924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:41417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:33481
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:32.713181 8924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:32.713418 8924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:32.718291 8929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:32.718382 8932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:32.718321 8930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:32.718350 8924 server_base.cc:1034] running on GCE node
I20260924 13:46:32.718685 8924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:32.718917 8924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:32.720054 8924 hybrid_clock.cc:648] HybridClock initialized: now 1790257592720045 us; error 32 us; skew 500 ppm
Sep 24 13:46:32 dist-test-slave-tfm9 krb5kdc[8911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257592, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:32.731019 8924 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:46:32.731530 8924 webserver.cc:460] Webserver started at http://127.19.53.254:41045/ using document root <none> and password file <none>
I20260924 13:46:32.731767 8924 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:32.731812 8924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:32.731902 8924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:32.732914 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "5b5e38319100468593d8ba0f22aea0d2"
format_stamp: "Formatted at 2026-09-24 13:46:32 on dist-test-slave-tfm9"
server_key: "0f188783c192a4da7b34926d8701b236"
server_key_iv: "297960aaea39830380ec943f59083eca"
server_key_version: "encryptionkey@0"
I20260924 13:46:32.733264 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "5b5e38319100468593d8ba0f22aea0d2"
format_stamp: "Formatted at 2026-09-24 13:46:32 on dist-test-slave-tfm9"
server_key: "0f188783c192a4da7b34926d8701b236"
server_key_iv: "297960aaea39830380ec943f59083eca"
server_key_version: "encryptionkey@0"
I20260924 13:46:32.734676 8924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:32.735412 8939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:32.735592 8924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:32.735678 8924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "5b5e38319100468593d8ba0f22aea0d2"
format_stamp: "Formatted at 2026-09-24 13:46:32 on dist-test-slave-tfm9"
server_key: "0f188783c192a4da7b34926d8701b236"
server_key_iv: "297960aaea39830380ec943f59083eca"
server_key_version: "encryptionkey@0"
I20260924 13:46:32.735747 8924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:32.755101 8924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:32.757069 8924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:32.757229 8924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:32.761441 8924 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:33481
I20260924 13:46:32.761502 8991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:33481 every 8 connection(s)
I20260924 13:46:32.761843 8924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:32.762709 8992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:32.764899 8992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2: Bootstrap starting.
I20260924 13:46:32.765365 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 8924
I20260924 13:46:32.765477 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:32.765601 8992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:32.765672 19671 external_mini_cluster.cc:1428] Setting key 2532ada9ebb88ef0511eb847ad2b981c
I20260924 13:46:32.765992 8992 log.cc:824] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:32.766788 8992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2: No bootstrap required, opened a new log
I20260924 13:46:32.769214 8992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5e38319100468593d8ba0f22aea0d2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33481 } }
I20260924 13:46:32.769348 8992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:32.769371 8992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5e38319100468593d8ba0f22aea0d2, State: Initialized, Role: FOLLOWER
I20260924 13:46:32.769505 8992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5e38319100468593d8ba0f22aea0d2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33481 } }
I20260924 13:46:32.769579 8992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:32.769601 8992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:32.769663 8992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:32.770365 8992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5e38319100468593d8ba0f22aea0d2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33481 } }
I20260924 13:46:32.770478 8992 leader_election.cc:302] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b5e38319100468593d8ba0f22aea0d2; no voters:
I20260924 13:46:32.770632 8992 leader_election.cc:288] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:32.770684 8997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:32.770932 8992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:32.770864 8997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [term 1 LEADER]: Becoming Leader. State: Replica: 5b5e38319100468593d8ba0f22aea0d2, State: Running, Role: LEADER
I20260924 13:46:32.771256 8997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5e38319100468593d8ba0f22aea0d2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33481 } }
I20260924 13:46:32.771704 8999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b5e38319100468593d8ba0f22aea0d2. Latest consensus state: current_term: 1 leader_uuid: "5b5e38319100468593d8ba0f22aea0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5e38319100468593d8ba0f22aea0d2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33481 } } }
I20260924 13:46:32.771797 8999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:32.772011 8998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b5e38319100468593d8ba0f22aea0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5e38319100468593d8ba0f22aea0d2" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33481 } } }
I20260924 13:46:32.772122 8998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:32.772295 9006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:32.772326 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:32.766199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54281 (local address 127.19.53.254:33481)
0924 13:46:32.766359 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:32.766369 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:32.766404 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:46:32.767016 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:32.767025 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:32.767085 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:32.767246 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:32.767415 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:32.768817 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:32.769811 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:32.770461 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:32.770570 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:32.771384 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:32.771389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:32.771390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:32.771612 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:32.771655 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:32.771671 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:32.771760 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:32.772045 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 13:46:32.772079 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260924 13:46:32.772814 9006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:32.773352 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:32.774516 9006 catalog_manager.cc:1347] Generated new cluster ID: e2039d4bc9584479a7009345962e7e9c
I20260924 13:46:32.774616 9006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:32.777668 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:32.774875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49427 (local address 127.19.53.254:33481)
0924 13:46:32.774909 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:32.774911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:32.775053 (+ 142us) server_negotiation.cc:407] Connection header received
0924 13:46:32.775124 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:32.775125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:32.775135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:32.775189 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:32.775342 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:32.775810 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:32.776538 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:32.776765 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:32.776840 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:32.777452 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:32.777453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:32.777453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:32.777457 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:32.777470 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:32.777474 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:32.777527 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:32.777585 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 13:46:32.777594 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 13:46:32.784353 9006 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:32.785003 9006 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:32.795758 9006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b5e38319100468593d8ba0f22aea0d2: Generated new TSK 0
I20260924 13:46:32.796022 9006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:32.805513 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 8924
W20260924 13:46:32.809984 8909 connection.cc:535] client connection to 127.19.53.254:33481 recv error: Network error: failed to read from TLS socket (remote: 127.19.53.254:33481): Connection reset by peer (error 104)
2026-09-24T13:46:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:32 dist-test-slave-tfm9 krb5kdc[9023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:32 dist-test-slave-tfm9 krb5kdc[9023](info): set up 2 sockets
Sep 24 13:46:32 dist-test-slave-tfm9 krb5kdc[9023](info): commencing operation
krb5kdc: starting...
W20260924 13:46:33.832002 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:33 dist-test-slave-tfm9 krb5kdc[9023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:46:33.952575 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:43661
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:43661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:34.038355 9036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:34.038546 9036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:34.038568 9036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:34.039713 9036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:34.039755 9036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:34.039769 9036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:34.039783 9036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:34.039820 9036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:34.041280 9036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:43661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:43661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:34.041523 9036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:34.041747 9036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:34.046527 9042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:34.046507 9041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:34.046561 9036 server_base.cc:1034] running on GCE node
W20260924 13:46:34.046527 9044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:34.046973 9036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:34.047283 9036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:34.048450 9036 hybrid_clock.cc:648] HybridClock initialized: now 1790257594048436 us; error 26 us; skew 500 ppm
Sep 24 13:46:34 dist-test-slave-tfm9 krb5kdc[9023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:34.050935 9036 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:46:34.051367 9036 webserver.cc:460] Webserver started at http://127.19.53.254:36623/ using document root <none> and password file <none>
I20260924 13:46:34.051590 9036 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:34.051658 9036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:34.051806 9036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:34.052850 9036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "9427ce2e8c664eb1b9d1f32a71d50bed"
format_stamp: "Formatted at 2026-09-24 13:46:34 on dist-test-slave-tfm9"
server_key: "3edf124270737761c80170eba02f9ae8"
server_key_iv: "429fe1cc327982b65590375e50f0fd75"
server_key_version: "encryptionkey@0"
I20260924 13:46:34.053208 9036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "9427ce2e8c664eb1b9d1f32a71d50bed"
format_stamp: "Formatted at 2026-09-24 13:46:34 on dist-test-slave-tfm9"
server_key: "3edf124270737761c80170eba02f9ae8"
server_key_iv: "429fe1cc327982b65590375e50f0fd75"
server_key_version: "encryptionkey@0"
I20260924 13:46:34.054674 9036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 13:46:34.055397 9051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:34.055559 9036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:34.055665 9036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "9427ce2e8c664eb1b9d1f32a71d50bed"
format_stamp: "Formatted at 2026-09-24 13:46:34 on dist-test-slave-tfm9"
server_key: "3edf124270737761c80170eba02f9ae8"
server_key_iv: "429fe1cc327982b65590375e50f0fd75"
server_key_version: "encryptionkey@0"
I20260924 13:46:34.055739 9036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:34.058995 9036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:34.060773 9036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:34.060936 9036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:34.065037 9036 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:43661
I20260924 13:46:34.065066 9103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:43661 every 8 connection(s)
I20260924 13:46:34.065387 9036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:34.066852 9104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:34.067108 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 9036
I20260924 13:46:34.067358 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:34.067713 19671 external_mini_cluster.cc:1428] Setting key 14f538685a595d4be22b5ac18a05b0c2
I20260924 13:46:34.070583 9104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed: Bootstrap starting.
I20260924 13:46:34.071686 9104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:34.072072 9104 log.cc:824] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:34.072299 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:34.068496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38187 (local address 127.19.53.254:43661)
0924 13:46:34.070066 (+ 1570us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:34.070077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:34.070114 (+ 37us) server_negotiation.cc:407] Connection header received
0924 13:46:34.070760 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:34.070769 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:34.070808 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:34.070973 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:34.071786 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:34.071790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:34.071791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:34.071795 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:34.071841 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:34.071857 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:34.071956 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:34.072020 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 13:46:34.072062 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1503,"thread_start_us":310,"threads_started":1}
I20260924 13:46:34.073021 9104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed: No bootstrap required, opened a new log
I20260924 13:46:34.073378 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:34.076025 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:34.074843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34329 (local address 127.19.53.254:43661)
0924 13:46:34.074881 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:34.074883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:34.074995 (+ 112us) server_negotiation.cc:407] Connection header received
0924 13:46:34.075064 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:34.075064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:34.075068 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:34.075138 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:34.075750 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:34.075751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:34.075751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:34.075755 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:34.075769 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:34.075773 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:34.075860 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:34.075923 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 13:46:34.075936 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:46:34.076177 9104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427ce2e8c664eb1b9d1f32a71d50bed" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43661 } }
I20260924 13:46:34.076377 9104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:34.076426 9104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9427ce2e8c664eb1b9d1f32a71d50bed, State: Initialized, Role: FOLLOWER
I20260924 13:46:34.076567 9104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427ce2e8c664eb1b9d1f32a71d50bed" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43661 } }
I20260924 13:46:34.076674 9104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:34.076714 9104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:34.076771 9104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:34.077595 9104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427ce2e8c664eb1b9d1f32a71d50bed" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43661 } }
I20260924 13:46:34.077731 9104 leader_election.cc:302] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9427ce2e8c664eb1b9d1f32a71d50bed; no voters:
I20260924 13:46:34.077898 9104 leader_election.cc:288] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:34.077972 9114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:34.078128 9114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [term 1 LEADER]: Becoming Leader. State: Replica: 9427ce2e8c664eb1b9d1f32a71d50bed, State: Running, Role: LEADER
I20260924 13:46:34.078183 9104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:34.078243 9114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427ce2e8c664eb1b9d1f32a71d50bed" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43661 } }
I20260924 13:46:34.078624 9115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9427ce2e8c664eb1b9d1f32a71d50bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427ce2e8c664eb1b9d1f32a71d50bed" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43661 } } }
I20260924 13:46:34.078742 9115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:34.078944 9120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:34.078619 9116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9427ce2e8c664eb1b9d1f32a71d50bed. Latest consensus state: current_term: 1 leader_uuid: "9427ce2e8c664eb1b9d1f32a71d50bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427ce2e8c664eb1b9d1f32a71d50bed" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 43661 } } }
I20260924 13:46:34.079233 9116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:34.079605 9120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:34.081185 9120 catalog_manager.cc:1347] Generated new cluster ID: d8b1a9b5b0e747c8a12a8e30388e87b2
I20260924 13:46:34.081248 9120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:34.119977 9120 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:34.120178 9120 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:34.122905 9120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9427ce2e8c664eb1b9d1f32a71d50bed: Generated new TSK 0
I20260924 13:46:34.123102 9120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:34.136138 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 9036
2026-09-24T13:46:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:34 dist-test-slave-tfm9 krb5kdc[9135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:34 dist-test-slave-tfm9 krb5kdc[9135](info): set up 2 sockets
Sep 24 13:46:34 dist-test-slave-tfm9 krb5kdc[9135](info): commencing operation
krb5kdc: starting...
W20260924 13:46:35.171629 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:35 dist-test-slave-tfm9 krb5kdc[9135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:46:35.286680 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:39353
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:45917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:39353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:35.363865 9148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:35.364053 9148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:35.364073 9148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:35.365130 9148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:35.365168 9148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:35.365180 9148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:35.365191 9148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:35.365203 9148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:35.366694 9148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:45917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:39353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:39353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:35.366952 9148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:35.367173 9148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:35.371850 9156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:35.371912 9148 server_base.cc:1034] running on GCE node
W20260924 13:46:35.371949 9154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:35.372077 9153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:35.372341 9148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:35.372630 9148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:35.373796 9148 hybrid_clock.cc:648] HybridClock initialized: now 1790257595373793 us; error 56 us; skew 500 ppm
Sep 24 13:46:35 dist-test-slave-tfm9 krb5kdc[9135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257595, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:35.376188 9148 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:46:35.376590 9148 webserver.cc:460] Webserver started at http://127.19.53.254:34105/ using document root <none> and password file <none>
I20260924 13:46:35.376835 9148 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:35.376904 9148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:35.377046 9148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:35.378139 9148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "65b83b3d8c60436ba0cfb3723cd1414f"
format_stamp: "Formatted at 2026-09-24 13:46:35 on dist-test-slave-tfm9"
server_key: "038b8a8b4e670aa541f3c48c98633c1a"
server_key_iv: "140a1d400a853c0376345ec27971e832"
server_key_version: "encryptionkey@0"
I20260924 13:46:35.378479 9148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "65b83b3d8c60436ba0cfb3723cd1414f"
format_stamp: "Formatted at 2026-09-24 13:46:35 on dist-test-slave-tfm9"
server_key: "038b8a8b4e670aa541f3c48c98633c1a"
server_key_iv: "140a1d400a853c0376345ec27971e832"
server_key_version: "encryptionkey@0"
I20260924 13:46:35.379771 9148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:35.380515 9163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:35.380681 9148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:35.380800 9148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "65b83b3d8c60436ba0cfb3723cd1414f"
format_stamp: "Formatted at 2026-09-24 13:46:35 on dist-test-slave-tfm9"
server_key: "038b8a8b4e670aa541f3c48c98633c1a"
server_key_iv: "140a1d400a853c0376345ec27971e832"
server_key_version: "encryptionkey@0"
I20260924 13:46:35.380874 9148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:35.397114 9148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:35.399067 9148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:35.399173 9148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:35.402989 9148 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:39353
I20260924 13:46:35.403024 9215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:39353 every 8 connection(s)
I20260924 13:46:35.403378 9148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:35.404247 9216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:35.406407 9216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f: Bootstrap starting.
I20260924 13:46:35.406981 9216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:35.407284 9216 log.cc:824] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:35.407938 9216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f: No bootstrap required, opened a new log
I20260924 13:46:35.409454 9216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b83b3d8c60436ba0cfb3723cd1414f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39353 } }
I20260924 13:46:35.409579 9216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:35.409601 9216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b83b3d8c60436ba0cfb3723cd1414f, State: Initialized, Role: FOLLOWER
I20260924 13:46:35.409780 9216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b83b3d8c60436ba0cfb3723cd1414f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39353 } }
I20260924 13:46:35.409859 9216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:35.409925 9216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:35.409977 9216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:35.410676 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 9148
I20260924 13:46:35.410810 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:35.410967 9216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b83b3d8c60436ba0cfb3723cd1414f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39353 } }
I20260924 13:46:35.411005 19671 external_mini_cluster.cc:1428] Setting key 29a1a0a1644d208f6bd9eea6b2491630
I20260924 13:46:35.411069 9216 leader_election.cc:302] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65b83b3d8c60436ba0cfb3723cd1414f; no voters:
I20260924 13:46:35.411231 9216 leader_election.cc:288] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:35.411242 9219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:35.411369 9219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [term 1 LEADER]: Becoming Leader. State: Replica: 65b83b3d8c60436ba0cfb3723cd1414f, State: Running, Role: LEADER
I20260924 13:46:35.411489 9216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:35.411497 9219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b83b3d8c60436ba0cfb3723cd1414f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39353 } }
I20260924 13:46:35.412070 9221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65b83b3d8c60436ba0cfb3723cd1414f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b83b3d8c60436ba0cfb3723cd1414f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39353 } } }
I20260924 13:46:35.412307 9221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:35.412225 9220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65b83b3d8c60436ba0cfb3723cd1414f. Latest consensus state: current_term: 1 leader_uuid: "65b83b3d8c60436ba0cfb3723cd1414f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b83b3d8c60436ba0cfb3723cd1414f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39353 } } }
I20260924 13:46:35.412500 9220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f [sys.catalog]: This master's current role is: LEADER
W20260924 13:46:35.413513 9237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:46:35.414112 9237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:46:35.414184 9230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:35.414350 9230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:35.415864 9230 catalog_manager.cc:1347] Generated new cluster ID: b418ebdc0e744d65aa140b81c961bd93
I20260924 13:46:35.415915 9230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:46:35 dist-test-slave-tfm9 krb5kdc[9135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:35.422392 9223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:35.411623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53441 (local address 127.19.53.254:39353)
0924 13:46:35.411789 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:35.411798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:35.411832 (+ 34us) server_negotiation.cc:407] Connection header received
0924 13:46:35.411946 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:35.411952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:35.411991 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:35.412122 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:35.413310 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:35.414599 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:35.415555 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:35.416133 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:35.416275 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:35.418650 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:35.418654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:35.418654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:35.418666 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:35.420847 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:35.421201 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:35.421204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:35.421204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:35.421301 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:35.421505 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:35.421506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:35.421506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:35.421737 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:35.421888 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:35.422056 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 13:46:35.422195 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260924 13:46:35.423381 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:35.429831 9230 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:35.430502 9230 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:35.435670 9223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:35.426224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46539 (local address 127.19.53.254:39353)
0924 13:46:35.426263 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:35.426265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:35.429952 (+ 3687us) server_negotiation.cc:407] Connection header received
0924 13:46:35.430035 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:35.430036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:35.430053 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:35.430119 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:35.430664 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:35.431143 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:35.431847 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:35.432036 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:35.432115 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:35.433228 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:35.433229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:35.433229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:35.433236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:35.434597 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:35.434941 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:35.434942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:35.434942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:35.434994 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:35.435182 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:35.435182 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:35.435183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:35.435283 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:35.435392 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:35.435483 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 13:46:35.435564 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:46:35.441814 9230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65b83b3d8c60436ba0cfb3723cd1414f: Generated new TSK 0
I20260924 13:46:35.442008 9230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:35.444470 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 9148
2026-09-24T13:46:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:35 dist-test-slave-tfm9 krb5kdc[9247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:35 dist-test-slave-tfm9 krb5kdc[9247](info): set up 2 sockets
Sep 24 13:46:35 dist-test-slave-tfm9 krb5kdc[9247](info): commencing operation
krb5kdc: starting...
W20260924 13:46:36.483755 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:36 dist-test-slave-tfm9 krb5kdc[9247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:46:36.600596 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33355
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:33687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:33355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:36.679723 9260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:36.679952 9260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:36.679988 9260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:36.681098 9260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:36.681169 9260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:36.681192 9260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:36.681229 9260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:36.681259 9260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:36.682679 9260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:33687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:33355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:33355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:36.682942 9260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:36.683153 9260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:36.688115 9268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:36.688127 9265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:36.688228 9266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:36.688153 9260 server_base.cc:1034] running on GCE node
I20260924 13:46:36.688571 9260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:36.688851 9260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:36.689970 9260 hybrid_clock.cc:648] HybridClock initialized: now 1790257596689960 us; error 41 us; skew 500 ppm
Sep 24 13:46:36 dist-test-slave-tfm9 krb5kdc[9247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257596, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:36.692381 9260 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:46:36.692772 9260 webserver.cc:460] Webserver started at http://127.19.53.254:41755/ using document root <none> and password file <none>
I20260924 13:46:36.692982 9260 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:36.693048 9260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:36.693193 9260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:36.694238 9260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "6cc95774a54f48589f5970a880d32b47"
format_stamp: "Formatted at 2026-09-24 13:46:36 on dist-test-slave-tfm9"
server_key: "fb77256945bdc996d8845309eb8b6630"
server_key_iv: "d9fe5cc9f97b16722b65bdd4a901abe9"
server_key_version: "encryptionkey@0"
I20260924 13:46:36.694626 9260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "6cc95774a54f48589f5970a880d32b47"
format_stamp: "Formatted at 2026-09-24 13:46:36 on dist-test-slave-tfm9"
server_key: "fb77256945bdc996d8845309eb8b6630"
server_key_iv: "d9fe5cc9f97b16722b65bdd4a901abe9"
server_key_version: "encryptionkey@0"
I20260924 13:46:36.695961 9260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:46:36.696748 9275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:36.696938 9260 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:36.697039 9260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "6cc95774a54f48589f5970a880d32b47"
format_stamp: "Formatted at 2026-09-24 13:46:36 on dist-test-slave-tfm9"
server_key: "fb77256945bdc996d8845309eb8b6630"
server_key_iv: "d9fe5cc9f97b16722b65bdd4a901abe9"
server_key_version: "encryptionkey@0"
I20260924 13:46:36.697124 9260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:36.706307 9260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:36.708302 9260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:36.708485 9260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:36.713109 9260 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:33355
I20260924 13:46:36.713143 9327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:33355 every 8 connection(s)
I20260924 13:46:36.713512 9260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:36.714432 9328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:36.715044 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 9260
I20260924 13:46:36.715178 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:36.715389 19671 external_mini_cluster.cc:1428] Setting key d15d0f436f97e3bcf2ae7923c1a14c1a
I20260924 13:46:36.717586 9328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47: Bootstrap starting.
I20260924 13:46:36.718274 9328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:36.718576 9328 log.cc:824] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:36.719281 9328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47: No bootstrap required, opened a new log
I20260924 13:46:36.721391 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:36.715913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33989 (local address 127.19.53.254:33355)
0924 13:46:36.716109 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:36.716118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:36.716148 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:46:36.716699 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:36.716708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:36.716780 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:36.716925 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:36.717157 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:36.718481 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:36.719294 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:36.719925 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:36.720039 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:36.720687 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:36.720692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:36.720693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:36.720922 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:36.720957 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:36.720971 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:36.721042 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:36.721103 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 13:46:36.721171 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260924 13:46:36.721985 9328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc95774a54f48589f5970a880d32b47" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33355 } }
I20260924 13:46:36.722147 9328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:36.722193 9328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc95774a54f48589f5970a880d32b47, State: Initialized, Role: FOLLOWER
I20260924 13:46:36.722333 9328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc95774a54f48589f5970a880d32b47" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33355 } }
I20260924 13:46:36.722419 9328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:36.722450 9328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:36.722527 9328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:36.722528 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:36.723095 9328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc95774a54f48589f5970a880d32b47" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33355 } }
I20260924 13:46:36.723181 9328 leader_election.cc:302] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cc95774a54f48589f5970a880d32b47; no voters:
I20260924 13:46:36.723361 9328 leader_election.cc:288] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:36.723393 9333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:36.723491 9333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [term 1 LEADER]: Becoming Leader. State: Replica: 6cc95774a54f48589f5970a880d32b47, State: Running, Role: LEADER
I20260924 13:46:36.723613 9328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:36.723629 9333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc95774a54f48589f5970a880d32b47" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33355 } }
I20260924 13:46:36.723995 9333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cc95774a54f48589f5970a880d32b47. Latest consensus state: current_term: 1 leader_uuid: "6cc95774a54f48589f5970a880d32b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc95774a54f48589f5970a880d32b47" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33355 } } }
I20260924 13:46:36.724067 9333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:36.724287 9335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cc95774a54f48589f5970a880d32b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc95774a54f48589f5970a880d32b47" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 33355 } } }
I20260924 13:46:36.724422 9335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:36.724579 9344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:36.725090 9344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:36.726688 9344 catalog_manager.cc:1347] Generated new cluster ID: 1d4bfbc3710c4a3cb1698cb369b1729d
I20260924 13:46:36.726754 9344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:36.727622 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:36.723955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53453 (local address 127.19.53.254:33355)
0924 13:46:36.723990 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:36.723992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:36.724151 (+ 159us) server_negotiation.cc:407] Connection header received
0924 13:46:36.724314 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:36.724316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:36.724341 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:36.724418 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:36.724550 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:36.725287 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:36.726288 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:36.726573 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:36.726638 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:36.727385 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:36.727386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:36.727387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:36.727390 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:36.727403 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:36.727406 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:36.727453 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:36.727512 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 13:46:36.727531 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:46:36.740923 9344 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:36.741629 9344 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:36.748281 9344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cc95774a54f48589f5970a880d32b47: Generated new TSK 0
I20260924 13:46:36.748471 9344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:36.752575 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 9260
2026-09-24T13:46:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:36 dist-test-slave-tfm9 krb5kdc[9359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:36 dist-test-slave-tfm9 krb5kdc[9359](info): set up 2 sockets
Sep 24 13:46:36 dist-test-slave-tfm9 krb5kdc[9359](info): commencing operation
krb5kdc: starting...
W20260924 13:46:37.775774 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:37 dist-test-slave-tfm9 krb5kdc[9359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:46:37.892518 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:35341
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:32921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:35341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:37.973165 9372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:37.973394 9372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:37.973433 9372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:37.974593 9372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:37.974661 9372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:37.974684 9372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:37.974712 9372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:37.974742 9372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:37.976104 9372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:32921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:35341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:35341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:37.976400 9372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:37.976615 9372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:37.981505 9378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:37.981510 9377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:37.981539 9372 server_base.cc:1034] running on GCE node
W20260924 13:46:37.981534 9380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:37.982107 9372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:37.982431 9372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:37.983582 9372 hybrid_clock.cc:648] HybridClock initialized: now 1790257597983551 us; error 45 us; skew 500 ppm
Sep 24 13:46:37 dist-test-slave-tfm9 krb5kdc[9359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257597, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:37.989504 9372 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:46:37.990013 9372 webserver.cc:460] Webserver started at http://127.19.53.254:34577/ using document root <none> and password file <none>
I20260924 13:46:37.990235 9372 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:37.990279 9372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:37.990422 9372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:37.991456 9372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "9c23a840617843caae9f6483fd21fcb8"
format_stamp: "Formatted at 2026-09-24 13:46:37 on dist-test-slave-tfm9"
server_key: "34fd844f92579a8674a75830ec826f16"
server_key_iv: "dc0e4f8e13e9c50ce57510140c9b6561"
server_key_version: "encryptionkey@0"
I20260924 13:46:37.991814 9372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "9c23a840617843caae9f6483fd21fcb8"
format_stamp: "Formatted at 2026-09-24 13:46:37 on dist-test-slave-tfm9"
server_key: "34fd844f92579a8674a75830ec826f16"
server_key_iv: "dc0e4f8e13e9c50ce57510140c9b6561"
server_key_version: "encryptionkey@0"
I20260924 13:46:37.993155 9372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:46:37.993860 9387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:37.994062 9372 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:37.994146 9372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "9c23a840617843caae9f6483fd21fcb8"
format_stamp: "Formatted at 2026-09-24 13:46:37 on dist-test-slave-tfm9"
server_key: "34fd844f92579a8674a75830ec826f16"
server_key_iv: "dc0e4f8e13e9c50ce57510140c9b6561"
server_key_version: "encryptionkey@0"
I20260924 13:46:37.994199 9372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:37.997318 9372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:37.999199 9372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:37.999306 9372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:38.003021 9372 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:35341
I20260924 13:46:38.003065 9439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:35341 every 8 connection(s)
I20260924 13:46:38.003391 9372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:38.004376 9440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:38.006651 9440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8: Bootstrap starting.
I20260924 13:46:38.006953 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 9372
I20260924 13:46:38.007079 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:38.007280 19671 external_mini_cluster.cc:1428] Setting key 1ed7ae65b87db0ac5e8d721ac6a8453c
I20260924 13:46:38.007395 9440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:38.007761 9440 log.cc:824] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:38.008497 9440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8: No bootstrap required, opened a new log
I20260924 13:46:38.009892 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:38.007803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60515 (local address 127.19.53.254:35341)
0924 13:46:38.008034 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:38.008041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:38.008067 (+ 26us) server_negotiation.cc:407] Connection header received
0924 13:46:38.008646 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:38.008655 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:38.008704 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:38.008849 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:38.009450 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:38.009454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:38.009454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:38.009458 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:38.009499 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:38.009512 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:38.009603 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:38.009633 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 13:46:38.009670 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260924 13:46:38.010910 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:38.011042 9440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c23a840617843caae9f6483fd21fcb8" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35341 } }
I20260924 13:46:38.011181 9440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:46:38.011243 9440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c23a840617843caae9f6483fd21fcb8, State: Initialized, Role: FOLLOWER
I20260924 13:46:38.011371 9440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c23a840617843caae9f6483fd21fcb8" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35341 } }
I20260924 13:46:38.011461 9440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:46:38.011505 9440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:46:38.011554 9440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:38.012135 9440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c23a840617843caae9f6483fd21fcb8" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35341 } }
I20260924 13:46:38.012243 9440 leader_election.cc:302] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c23a840617843caae9f6483fd21fcb8; no voters:
I20260924 13:46:38.012388 9440 leader_election.cc:288] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:46:38.012477 9449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:38.012643 9440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:38.012646 9449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [term 1 LEADER]: Becoming Leader. State: Replica: 9c23a840617843caae9f6483fd21fcb8, State: Running, Role: LEADER
I20260924 13:46:38.012837 9449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c23a840617843caae9f6483fd21fcb8" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35341 } }
I20260924 13:46:38.013270 9452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c23a840617843caae9f6483fd21fcb8. Latest consensus state: current_term: 1 leader_uuid: "9c23a840617843caae9f6483fd21fcb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c23a840617843caae9f6483fd21fcb8" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35341 } } }
I20260924 13:46:38.013420 9452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:38.013393 9451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c23a840617843caae9f6483fd21fcb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c23a840617843caae9f6483fd21fcb8" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 35341 } } }
I20260924 13:46:38.013573 9451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:38.013707 9459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:38.013808 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:38.012390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53347 (local address 127.19.53.254:35341)
0924 13:46:38.012426 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:38.012428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:38.012596 (+ 168us) server_negotiation.cc:407] Connection header received
0924 13:46:38.012760 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:38.012761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:38.012765 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:38.012817 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:38.013354 (+ 537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:38.013355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:38.013355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:46:38.013359 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:38.013372 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:46:38.013377 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:38.013425 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:38.013695 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 13:46:38.013710 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:46:38.014320 9459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:38.015780 9459 catalog_manager.cc:1347] Generated new cluster ID: 0b4eaffdf6604b4c8118357ae594158f
I20260924 13:46:38.015829 9459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:38.025008 9459 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:38.025159 9459 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:38.028946 9459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c23a840617843caae9f6483fd21fcb8: Generated new TSK 0
I20260924 13:46:38.029119 9459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:38.035247 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 9372
2026-09-24T13:46:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1282 ms)
[----------] 18 tests from AuthTokenIssuingTest (23760 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:38 dist-test-slave-tfm9 krb5kdc[9471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:38 dist-test-slave-tfm9 krb5kdc[9471](info): set up 2 sockets
Sep 24 13:46:38 dist-test-slave-tfm9 krb5kdc[9471](info): commencing operation
krb5kdc: starting...
W20260924 13:46:39.071743 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:46:39.191269 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:39473
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:39473,127.19.53.253:34105,127.19.53.252:41769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:39.272533 9484 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:46:39.272821 9484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:39.272853 9484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:39.272902 9484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:39.274042 9484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:39.274104 9484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:39.274124 9484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:39.274143 9484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:39.274168 9484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:39.275588 9484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:39473,127.19.53.253:34105,127.19.53.252:41769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:39473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:39.275857 9484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:39.276064 9484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:39.280864 9489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:39.280974 9484 server_base.cc:1034] running on GCE node
W20260924 13:46:39.280881 9490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:39.280872 9492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:39.281313 9484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:39.281608 9484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:39.282747 9484 hybrid_clock.cc:648] HybridClock initialized: now 1790257599282726 us; error 39 us; skew 500 ppm
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:39.285148 9484 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:46:39.285591 9484 webserver.cc:460] Webserver started at http://127.19.53.254:45177/ using document root <none> and password file <none>
I20260924 13:46:39.285777 9484 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:39.285828 9484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:39.285983 9484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:39.286993 9484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "acc4f8d231364aa59e1e72b30f9a59a7"
format_stamp: "Formatted at 2026-09-24 13:46:39 on dist-test-slave-tfm9"
server_key: "ae3993ce4b6d755c7e8a0dbddb156b6a"
server_key_iv: "b4707f7169eb853f76fd9d46fc21bb0f"
server_key_version: "encryptionkey@0"
I20260924 13:46:39.287295 9484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "acc4f8d231364aa59e1e72b30f9a59a7"
format_stamp: "Formatted at 2026-09-24 13:46:39 on dist-test-slave-tfm9"
server_key: "ae3993ce4b6d755c7e8a0dbddb156b6a"
server_key_iv: "b4707f7169eb853f76fd9d46fc21bb0f"
server_key_version: "encryptionkey@0"
I20260924 13:46:39.288511 9484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:46:39.289192 9499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:39.289359 9484 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:46:39.289459 9484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "acc4f8d231364aa59e1e72b30f9a59a7"
format_stamp: "Formatted at 2026-09-24 13:46:39 on dist-test-slave-tfm9"
server_key: "ae3993ce4b6d755c7e8a0dbddb156b6a"
server_key_iv: "b4707f7169eb853f76fd9d46fc21bb0f"
server_key_version: "encryptionkey@0"
I20260924 13:46:39.289532 9484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:39.298588 9484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:39.300699 9484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:39.300870 9484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:39.305083 9484 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:39473
I20260924 13:46:39.305124 9551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:39473 every 8 connection(s)
I20260924 13:46:39.305480 9484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:39.305763 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 9484
I20260924 13:46:39.305879 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:39.306123 19671 external_mini_cluster.cc:1428] Setting key 8413b9e461475f7654a02797f13f4140
I20260924 13:46:39.306521 9552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:39.308506 9552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.53.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.253@KRBTEST.COM" created.
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:39.320997 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.309236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:39473 (local address 127.0.0.1:46930)
0924 13:46:39.309420 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.309458 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:39.310054 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:39.310352 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:39.310355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:39.310630 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:39.310808 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.310813 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.311786 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.311787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.312429 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.312431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.312913 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.312914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.312974 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.313488 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:39.313495 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:39.316818 (+ 3323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:39.319013 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.319016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.319018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.319299 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.319557 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.319558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.319558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.319649 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.320310 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:39.320319 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:39.320400 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 13:46:39.320502 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 13:46:39.320687 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260924 13:46:39.321055 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.309283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.19.53.254:39473)
0924 13:46:39.309433 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.309442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.309659 (+ 217us) server_negotiation.cc:407] Connection header received
0924 13:46:39.310222 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.310224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.310255 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.310346 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.310924 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.311687 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.312524 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.312817 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.312929 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.316918 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.316924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.316925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.316932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.318925 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.319382 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.319385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.319385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.319476 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.319726 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.319727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.319727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.320183 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.320308 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.320569 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.320834 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260924 13:46:39.324803 9552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:46:39.325220 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.325046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:34105 (local address 127.0.0.1:51464)
0924 13:46:39.325121 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.325172 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:34105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 13:46:39.325359 9503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.53.253:34105: connect: Connection refused (error 111)
I20260924 13:46:39.325817 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.325671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:34105 (local address 127.0.0.1:51474)
0924 13:46:39.325766 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.325790 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:34105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 13:46:39.325965 9552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.253:34105: Network error: Client connection negotiation failed: client connection to 127.19.53.253:34105: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.253.keytab.
Entry for principal kudu/127.19.53.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.253.keytab.
I20260924 13:46:39.333177 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.253:34105
--webserver_interface=127.19.53.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.253.keytab
--principal=kudu/127.19.53.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:39473,127.19.53.253:34105,127.19.53.252:41769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:46:39.375200 9552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } attempt: 1
I20260924 13:46:39.375702 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.375563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:34105 (local address 127.0.0.1:51484)
0924 13:46:39.375633 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.375672 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:34105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 13:46:39.376048 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.375951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:34105 (local address 127.0.0.1:51496)
0924 13:46:39.375996 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.376024 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:34105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 13:46:39.376134 9552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.253:34105: Network error: Client connection negotiation failed: client connection to 127.19.53.253:34105: connect: Connection refused (error 111)
W20260924 13:46:39.413551 9559 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:46:39.413743 9559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:39.413758 9559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:39.413777 9559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:39.414928 9559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:39.414970 9559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:39.414983 9559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:39.414995 9559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:39.415046 9559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:39.416488 9559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:39473,127.19.53.253:34105,127.19.53.252:41769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.253.keytab
--principal=kudu/127.19.53.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.253:34105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.53.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:39.416726 9559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:39.416957 9559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:39.421619 9567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:39.421623 9564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:39.421617 9565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:39.421677 9559 server_base.cc:1034] running on GCE node
I20260924 13:46:39.422120 9559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:39.422438 9559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:39.423602 9559 hybrid_clock.cc:648] HybridClock initialized: now 1790257599423597 us; error 35 us; skew 500 ppm
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:39.426074 9559 init.cc:377] Logged in from keytab as kudu/127.19.53.253@KRBTEST.COM (short username kudu)
I20260924 13:46:39.426573 9559 webserver.cc:460] Webserver started at http://127.19.53.253:42561/ using document root <none> and password file <none>
I20260924 13:46:39.426810 9559 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:39.426898 9559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:39.427029 9559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:39.428035 9559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/data/instance:
uuid: "905fd32ffae84079a39ccde4b3c168ff"
format_stamp: "Formatted at 2026-09-24 13:46:39 on dist-test-slave-tfm9"
server_key: "59cf11d9a0bdf2c82cfb70af0526faf9"
server_key_iv: "86fc4b7e194b417c467904f94482ba07"
server_key_version: "encryptionkey@0"
I20260924 13:46:39.428403 9559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/wal/instance:
uuid: "905fd32ffae84079a39ccde4b3c168ff"
format_stamp: "Formatted at 2026-09-24 13:46:39 on dist-test-slave-tfm9"
server_key: "59cf11d9a0bdf2c82cfb70af0526faf9"
server_key_iv: "86fc4b7e194b417c467904f94482ba07"
server_key_version: "encryptionkey@0"
I20260924 13:46:39.429714 9559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:39.430555 9574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:39.430733 9559 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:39.430817 9559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/wal
uuid: "905fd32ffae84079a39ccde4b3c168ff"
format_stamp: "Formatted at 2026-09-24 13:46:39 on dist-test-slave-tfm9"
server_key: "59cf11d9a0bdf2c82cfb70af0526faf9"
server_key_iv: "86fc4b7e194b417c467904f94482ba07"
server_key_version: "encryptionkey@0"
I20260924 13:46:39.430904 9559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:39.444272 9552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } attempt: 2
I20260924 13:46:39.444689 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.444563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:34105 (local address 127.0.0.1:51510)
0924 13:46:39.444623 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.444662 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:34105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 13:46:39.445024 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.444929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:34105 (local address 127.0.0.1:51514)
0924 13:46:39.444973 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.445000 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:34105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 13:46:39.445104 9552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.253:34105: Network error: Client connection negotiation failed: client connection to 127.19.53.253:34105: connect: Connection refused (error 111)
I20260924 13:46:39.450174 9559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:39.452054 9559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:39.452211 9559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:39.456300 9559 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.253:34105
I20260924 13:46:39.456332 9626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.253:34105 every 8 connection(s)
I20260924 13:46:39.456689 9559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/data/info.pb
I20260924 13:46:39.457597 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 9559
I20260924 13:46:39.457739 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-1/wal/instance
I20260924 13:46:39.457682 9627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:39.457996 19671 external_mini_cluster.cc:1428] Setting key 73e53bf38a97d8e206d15a852f0cd0d3
I20260924 13:46:39.459578 9627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.53.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.252@KRBTEST.COM" created.
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.253@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:39.472311 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.460350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.19.53.254:39473)
0924 13:46:39.460422 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.460425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.460689 (+ 264us) server_negotiation.cc:407] Connection header received
0924 13:46:39.461331 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.461332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.461343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.461383 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.462273 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.462705 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.464178 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.464355 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.464408 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.469461 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.469462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.469463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.469470 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.470890 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.471300 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.471301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.471302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.471364 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.471630 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.471631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.471631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.471756 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.471825 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.472116 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.472224 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:46:39.472519 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.460344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:39473 (local address 127.0.0.1:46944)
0924 13:46:39.460545 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.460588 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:39.461226 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:39.461448 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:39.461452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:39.461837 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:39.462136 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.462142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.462801 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.462802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.464090 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.464093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.464447 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.464448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.464592 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.465047 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:39.465064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:39.469365 (+ 4301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:39.470993 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.471000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.471002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.471224 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.471436 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.471437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.471438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.471553 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.471830 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:39.471837 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:39.471967 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 13:46:39.472097 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 13:46:39.472255 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260924 13:46:39.473111 9627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.253@KRBTEST.COM for kudu/127.19.53.253@KRBTEST.COM
I20260924 13:46:39.481496 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.473510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51522 (local address 127.19.53.253:34105)
0924 13:46:39.473662 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.473665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.473681 (+ 16us) server_negotiation.cc:407] Connection header received
0924 13:46:39.473699 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.473701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.473711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.473792 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.474119 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.474793 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.475478 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.475792 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.475881 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.478736 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.478743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.478743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.478749 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.480464 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.480793 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.480795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.480796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.480845 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.481026 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.481026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.481027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.481150 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.481208 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.481364 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.481427 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260924 13:46:39.481550 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.473418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:34105 (local address 127.0.0.1:51522)
0924 13:46:39.473468 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.473478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:39.473514 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:39.473793 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:39.473793 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:39.473988 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:39.474047 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.474049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.474893 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.474894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.475389 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.475390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.475880 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.475881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.475943 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.476407 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:39.476411 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:39.478652 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:39.480536 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.480537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.480537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.480695 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.480905 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.480906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.480906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.480972 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.481206 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:39.481206 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:39.481241 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:46:39.481345 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 13:46:39.481471 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":32}
I20260924 13:46:39.482183 9627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.252.keytab.
Entry for principal kudu/127.19.53.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.252.keytab.
I20260924 13:46:39.482563 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.482436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:41769 (local address 127.0.0.1:43840)
0924 13:46:39.482481 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.482522 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:41769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 13:46:39.482684 9578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.53.252:41769: connect: Connection refused (error 111)
I20260924 13:46:39.483114 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.483030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:41769 (local address 127.0.0.1:43850)
0924 13:46:39.483079 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.483101 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:41769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 13:46:39.483242 9627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:41769: Network error: Client connection negotiation failed: client connection to 127.19.53.252:41769: connect: Connection refused (error 111)
I20260924 13:46:39.483518 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.252:41769
--webserver_interface=127.19.53.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.252.keytab
--principal=kudu/127.19.53.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:39473,127.19.53.253:34105,127.19.53.252:41769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:46:39.532550 9627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } attempt: 1
I20260924 13:46:39.533064 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.532910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:41769 (local address 127.0.0.1:43864)
0924 13:46:39.532976 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.533025 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:41769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 13:46:39.533545 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.533451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:41769 (local address 127.0.0.1:43868)
0924 13:46:39.533493 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.533522 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:41769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 13:46:39.533668 9627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:41769: Network error: Client connection negotiation failed: client connection to 127.19.53.252:41769: connect: Connection refused (error 111)
I20260924 13:46:39.536247 9552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } attempt: 3
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for kudu/127.19.53.253@KRBTEST.COM
I20260924 13:46:39.543059 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.536610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.19.53.253:34105)
0924 13:46:39.536659 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.536661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.536671 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:46:39.536751 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.536751 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.536762 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.536817 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.537142 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.537637 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.538422 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.538649 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.538735 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.540502 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.540503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.540503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.540510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.541886 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.542252 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.542254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.542254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.542313 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.542540 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.542541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.542542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.542652 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.542728 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.542894 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.542967 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 13:46:39.543066 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.536550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:34105 (local address 127.0.0.1:51536)
0924 13:46:39.536604 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.536615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:39.536667 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:39.536824 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:39.536824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:39.536989 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:39.537055 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.537058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.537734 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.537736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.538332 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.538335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.538736 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.538737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.538818 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.539270 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:39.539275 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:39.540418 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:39.542001 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.542003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.542003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.542173 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.542392 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.542393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.542394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.542455 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.542721 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:39.542722 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:39.542785 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 13:46:39.542869 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 13:46:39.542967 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 13:46:39.543560 9552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:46:39.543902 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.543789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:41769 (local address 127.0.0.1:43870)
0924 13:46:39.543854 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.543882 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:41769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 13:46:39.544196 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.544141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:41769 (local address 127.0.0.1:43876)
0924 13:46:39.544163 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.544181 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:41769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 13:46:39.544294 9552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:41769: Network error: Client connection negotiation failed: client connection to 127.19.53.252:41769: connect: Connection refused (error 111)
W20260924 13:46:39.566562 9634 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:46:39.566758 9634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:39.566774 9634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:39.566794 9634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:39.567937 9634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:39.567975 9634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:39.567991 9634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:39.568002 9634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:39.568014 9634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:39.569410 9634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:39473,127.19.53.253:34105,127.19.53.252:41769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.252.keytab
--principal=kudu/127.19.53.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.252:41769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.53.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:39.569585 9634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:39.569761 9634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:39.574493 9642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:39.574535 9640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:39.574610 9634 server_base.cc:1034] running on GCE node
W20260924 13:46:39.574517 9639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:39.574951 9634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:39.575206 9634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:39.575421 9552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } attempt: 1
I20260924 13:46:39.575758 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.575655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:41769 (local address 127.0.0.1:43880)
0924 13:46:39.575705 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.575734 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:41769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 13:46:39.576061 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.575986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:41769 (local address 127.0.0.1:43892)
0924 13:46:39.576025 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.576046 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:41769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 13:46:39.576157 9552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:41769: Network error: Client connection negotiation failed: client connection to 127.19.53.252:41769: connect: Connection refused (error 111)
I20260924 13:46:39.576373 9634 hybrid_clock.cc:648] HybridClock initialized: now 1790257599576353 us; error 36 us; skew 500 ppm
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:39.578935 9634 init.cc:377] Logged in from keytab as kudu/127.19.53.252@KRBTEST.COM (short username kudu)
I20260924 13:46:39.579412 9634 webserver.cc:460] Webserver started at http://127.19.53.252:38795/ using document root <none> and password file <none>
I20260924 13:46:39.579646 9634 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:39.579689 9634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:39.579825 9634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:39.580847 9634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/data/instance:
uuid: "a81b45a849a14dde8e00aecae2249bcf"
format_stamp: "Formatted at 2026-09-24 13:46:39 on dist-test-slave-tfm9"
server_key: "78ffb7a1dba5382c50ae931772236eed"
server_key_iv: "a2afdf889799b849c3406c81a2187e10"
server_key_version: "encryptionkey@0"
I20260924 13:46:39.581205 9634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/wal/instance:
uuid: "a81b45a849a14dde8e00aecae2249bcf"
format_stamp: "Formatted at 2026-09-24 13:46:39 on dist-test-slave-tfm9"
server_key: "78ffb7a1dba5382c50ae931772236eed"
server_key_iv: "a2afdf889799b849c3406c81a2187e10"
server_key_version: "encryptionkey@0"
I20260924 13:46:39.582506 9634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:46:39.583191 9649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:39.583491 9634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:46:39.583587 9634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/wal
uuid: "a81b45a849a14dde8e00aecae2249bcf"
format_stamp: "Formatted at 2026-09-24 13:46:39 on dist-test-slave-tfm9"
server_key: "78ffb7a1dba5382c50ae931772236eed"
server_key_iv: "a2afdf889799b849c3406c81a2187e10"
server_key_version: "encryptionkey@0"
I20260924 13:46:39.583673 9634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:39.590910 9634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:39.592774 9634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:39.592953 9634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:39.596787 9634 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.252:41769
I20260924 13:46:39.596815 9701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.252:41769 every 8 connection(s)
I20260924 13:46:39.597136 9634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/data/info.pb
I20260924 13:46:39.597934 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 9634
I20260924 13:46:39.598031 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257335418480-19671-0/minicluster-data/master-2/wal/instance
I20260924 13:46:39.598230 19671 external_mini_cluster.cc:1428] Setting key 52d59d8bf18f12067a84b93d580944c7
I20260924 13:46:39.598306 9702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:39.600589 9702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:46:39.601841 9627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } attempt: 2
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.252@KRBTEST.COM
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.252@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.253@KRBTEST.COM for kudu/127.19.53.252@KRBTEST.COM
I20260924 13:46:39.609825 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.601231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:39473 (local address 127.0.0.1:46952)
0924 13:46:39.601400 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.601441 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:39.601542 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:39.601679 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:39.601682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:39.602044 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:39.602114 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.602116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.603032 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.603033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.603856 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.603858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.604342 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.604344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.604428 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.604882 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:39.604890 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:39.606353 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:39.608422 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.608427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.608429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.608721 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.608930 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.608931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.608931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.608993 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.609326 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:39.609333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:39.609395 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 13:46:39.609501 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 13:46:39.609620 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":23,"thread_start_us":104,"threads_started":1}
I20260924 13:46:39.609890 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.601252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.19.53.254:39473)
0924 13:46:39.601318 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.601320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.601539 (+ 219us) server_negotiation.cc:407] Connection header received
0924 13:46:39.601605 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.601606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.601615 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.601663 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.602216 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.602924 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.603941 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.604227 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.604293 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.606446 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.606448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.606448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.606475 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.608338 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.608795 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.608797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.608797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.608855 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.609093 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.609095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.609095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.609235 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.609276 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.609705 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.609779 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:46:39.609825 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.598695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.19.53.252:41769)
0924 13:46:39.598921 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.598931 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.598966 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:46:39.599482 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.599490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.599545 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.599685 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.600257 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.601518 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.602641 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.603119 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.603240 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.605803 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.605810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.605811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.605817 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.607989 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.608329 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.608332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.608332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.608418 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.608625 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.608626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.608627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.608911 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.609047 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.609428 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.609563 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
I20260924 13:46:39.610180 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.602115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:41769 (local address 127.0.0.1:43904)
0924 13:46:39.602154 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.602165 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:39.602204 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:39.602488 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:39.602489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:39.602658 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:39.602701 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.602703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.603575 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.603576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.604308 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.604311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.604694 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.604695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.604754 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.605067 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:39.605072 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:39.606738 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:39.609104 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.609105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.609105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.609307 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.609519 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.609520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.609520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.609576 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.609873 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:39.609874 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:39.609937 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 13:46:39.609988 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 13:46:39.610072 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 13:46:39.610605 9702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:46:39.610656 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.602170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43904 (local address 127.19.53.252:41769)
0924 13:46:39.602322 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.602323 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.602330 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:46:39.602347 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.602348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.602368 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.602481 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.602781 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.603496 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.604369 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.604615 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.604678 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.606806 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.606807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.606808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.606813 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.609000 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.609407 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.609408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.609408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.609468 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.609661 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.609662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.609662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.609792 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.610376 (+ 584us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.610416 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.610500 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":109,"threads_started":1}
I20260924 13:46:39.612182 9627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff: Bootstrap starting.
I20260924 13:46:39.613011 9627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:39.613312 9627 log.cc:824] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:39.614108 9627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff: No bootstrap required, opened a new log
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.252@KRBTEST.COM for kudu/127.19.53.253@KRBTEST.COM
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:39.616026 9627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } }
I20260924 13:46:39.616161 9627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905fd32ffae84079a39ccde4b3c168ff, State: Initialized, Role: FOLLOWER
I20260924 13:46:39.616269 9627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } }
I20260924 13:46:39.616600 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } } }
I20260924 13:46:39.616703 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:46:39.616611 9627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:39.618881 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.610892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.19.53.253:34105)
0924 13:46:39.610928 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.610930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.611028 (+ 98us) server_negotiation.cc:407] Connection header received
0924 13:46:39.611110 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.611111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.611127 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.611172 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.611488 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.612008 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.612876 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.613166 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.613246 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.615249 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.615250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.615251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.615258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.617494 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.617980 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.617981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.617982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.618042 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.618284 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.618285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.618286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.618458 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.618503 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.618721 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.618801 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 13:46:39.618906 9720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:46:39.618942 9720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:46:39.619160 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.610847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.19.53.254:39473)
0924 13:46:39.610959 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.610961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.611048 (+ 87us) server_negotiation.cc:407] Connection header received
0924 13:46:39.611081 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.611082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.611091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.611138 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.611624 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.612294 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.613148 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.613703 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.613767 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.616074 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.616076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.616076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.616083 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.617715 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.618111 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.618113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.618113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.618166 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.618370 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.618371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.618371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.618505 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.618654 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.619007 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.619076 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 13:46:39.619545 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.610856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:34105 (local address 127.0.0.1:51552)
0924 13:46:39.610979 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.610988 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:39.611031 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:39.611215 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:39.611216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:39.611349 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:39.611403 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.611405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.612084 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.612085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.612792 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.612794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.613244 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.613245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.613313 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.613706 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:39.613711 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:39.615145 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:39.617635 (+ 2490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.617636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.617637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.617857 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.618134 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.618135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.618135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.618199 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.618554 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:39.618555 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:39.618594 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 13:46:39.619414 (+ 820us) client_negotiation.cc:240] Negotiation successful
0924 13:46:39.619488 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260924 13:46:39.619928 9702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:46:39.619099 9559 master_runner.cc:186] Error getting master registration for 127.19.53.254:39473: OK, instance_id { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" instance_seqno: 1790257599303880 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:46:39.621428 9559 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.252@KRBTEST.COM for kudu/127.19.53.252@KRBTEST.COM
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.253@KRBTEST.COM
I20260924 13:46:39.629040 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.621899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.19.53.253:34105)
0924 13:46:39.621950 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.621952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.621959 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:46:39.622012 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.622012 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.622021 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.622072 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.622495 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.622912 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.623526 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.623717 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.623797 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.626397 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.626398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.626399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.626405 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.627892 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.628272 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.628273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.628273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.628324 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.628547 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.628548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.628548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.628650 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.628819 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.628861 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.628930 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 13:46:39.629484 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.621495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.19.53.252:41769)
0924 13:46:39.621553 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.621555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.621564 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:46:39.621582 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.621583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.621592 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.621654 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.621966 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.622649 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.623524 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.623814 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.623878 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.626292 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.626294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.626295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.626311 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.627909 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.628277 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.628278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.628278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.628327 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.628535 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.628536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.628537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.628707 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.628775 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.629324 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.629412 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:46:39.629557 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.620176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:41769 (local address 127.0.0.1:43920)
0924 13:46:39.620210 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.620220 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:39.620258 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:39.621675 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:39.621676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:39.621816 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:39.621847 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.621849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.622732 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.622733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.623443 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.623445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.623874 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.623875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.623931 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.624395 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:39.624400 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:39.626205 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:39.627994 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.627994 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.627995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.628188 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.628394 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.628395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.628396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.628461 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.628798 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:39.628798 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:39.628832 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 13:46:39.628893 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 13:46:39.628984 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 13:46:39.631122 9702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf: Bootstrap starting.
I20260924 13:46:39.631845 9702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:39.632143 9702 log.cc:824] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:39.632741 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:39.632923 9702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf: No bootstrap required, opened a new log
I20260924 13:46:39.634003 9526 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "acc4f8d231364aa59e1e72b30f9a59a7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46960
I20260924 13:46:39.635146 9702 raft_consensus.cc:348] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } }
I20260924 13:46:39.635293 9702 raft_consensus.cc:729] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a81b45a849a14dde8e00aecae2249bcf, State: Initialized, Role: FOLLOWER
I20260924 13:46:39.635380 9702 consensus_queue.cc:260] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } }
I20260924 13:46:39.635705 9702 sys_catalog.cc:564] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:39.635665 9723 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } } }
I20260924 13:46:39.635833 9723 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf [sys.catalog]: This master's current role is: FOLLOWER
W20260924 13:46:39.637882 9734 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:46:39.638002 9734 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:46:39.637718 9634 master_runner.cc:186] Error getting master registration for 127.19.53.254:39473: OK, instance_id { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" instance_seqno: 1790257599303880 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:46:39.639952 9634 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:46:39.651290 9552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } attempt: 2
Sep 24 13:46:39 dist-test-slave-tfm9 krb5kdc[9471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257599, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for kudu/127.19.53.252@KRBTEST.COM
I20260924 13:46:39.659283 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.651575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:41769 (local address 127.0.0.1:43930)
0924 13:46:39.651630 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:39.651640 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:39.651676 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:39.651798 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:39.651799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:39.651971 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:39.652020 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.652022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.652990 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.652991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.653817 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:39.653820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.654328 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.654329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:39.654389 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.654877 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:39.654881 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:39.656374 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:39.658255 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.658256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.658257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.658464 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.658674 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:39.658675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:39.658675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:39.658731 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:39.658989 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:39.658990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:39.659033 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 13:46:39.659097 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 13:46:39.659180 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 13:46:39.659570 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.651638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.19.53.252:41769)
0924 13:46:39.651677 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.651679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.651699 (+ 20us) server_negotiation.cc:407] Connection header received
0924 13:46:39.651723 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.651724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.651733 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.651795 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.652096 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.652876 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.653933 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.654237 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.654316 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.656462 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.656465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.656465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.656472 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.658176 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.658539 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.658540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.658541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.658601 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.658795 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.658796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.658796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.658918 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.658991 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.659144 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.659237 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:46:39.661293 9552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7: Bootstrap starting.
I20260924 13:46:39.662026 9552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:39.662374 9552 log.cc:824] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:39.663268 9552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7: No bootstrap required, opened a new log
I20260924 13:46:39.664639 9552 raft_consensus.cc:348] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } }
I20260924 13:46:39.664784 9552 raft_consensus.cc:729] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc4f8d231364aa59e1e72b30f9a59a7, State: Initialized, Role: FOLLOWER
I20260924 13:46:39.664865 9552 consensus_queue.cc:260] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } }
I20260924 13:46:39.665162 9552 sys_catalog.cc:564] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:39.665220 9526 raft_consensus.cc:482] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 13:46:39.665287 9526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:39.666220 9526 raft_consensus.cc:504] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } }
I20260924 13:46:39.666868 9601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "905fd32ffae84079a39ccde4b3c168ff"
I20260924 13:46:39.666882 9676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a81b45a849a14dde8e00aecae2249bcf"
I20260924 13:46:39.667011 9601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:39.667019 9676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:39.666483 9526 leader_election.cc:288] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [CANDIDATE]: Term 1 election: Requested vote from peers 905fd32ffae84079a39ccde4b3c168ff (127.19.53.253:34105), a81b45a849a14dde8e00aecae2249bcf (127.19.53.252:41769)
I20260924 13:46:39.667856 9676 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc4f8d231364aa59e1e72b30f9a59a7 in term 1.
I20260924 13:46:39.667856 9601 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc4f8d231364aa59e1e72b30f9a59a7 in term 1.
I20260924 13:46:39.668082 9503 leader_election.cc:302] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 905fd32ffae84079a39ccde4b3c168ff, acc4f8d231364aa59e1e72b30f9a59a7; no voters:
I20260924 13:46:39.665167 9737 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } } }
I20260924 13:46:39.668346 9737 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:46:39.668429 9737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:39.668529 9737 raft_consensus.cc:686] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [term 1 LEADER]: Becoming Leader. State: Replica: acc4f8d231364aa59e1e72b30f9a59a7, State: Running, Role: LEADER
I20260924 13:46:39.668638 9737 consensus_queue.cc:237] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } }
I20260924 13:46:39.669066 9737 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acc4f8d231364aa59e1e72b30f9a59a7. Latest consensus state: current_term: 1 leader_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } } }
I20260924 13:46:39.669211 9737 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:39.669430 9755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:39.669914 9755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:39.670249 9484 master_runner.cc:263] No leader master found from master acc4f8d231364aa59e1e72b30f9a59a7
I20260924 13:46:39.670300 9484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:46:39.672205 9676 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf [term 1 FOLLOWER]: Refusing update from remote peer acc4f8d231364aa59e1e72b30f9a59a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:39.672223 9601 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff [term 1 FOLLOWER]: Refusing update from remote peer acc4f8d231364aa59e1e72b30f9a59a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:39.672478 9737 consensus_queue.cc:1035] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:39.672616 9737 consensus_queue.cc:1035] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:39.673768 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader acc4f8d231364aa59e1e72b30f9a59a7. Latest consensus state: current_term: 1 leader_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } } }
I20260924 13:46:39.673854 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:46:39.674641 9723 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader acc4f8d231364aa59e1e72b30f9a59a7. Latest consensus state: current_term: 1 leader_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } } }
I20260924 13:46:39.674721 9723 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:46:39.679139 9737 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } } }
I20260924 13:46:39.679229 9737 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:39.679260 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } } }
I20260924 13:46:39.679337 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:46:39.679445 9757 mvcc.cc:205] Tried to move back new op lower bound from 7332895128255283200 to 7332895128243560448. Current Snapshot: MvccSnapshot[applied={T|T < 7332895128255283200}]
I20260924 13:46:39.679517 9758 mvcc.cc:205] Tried to move back new op lower bound from 7332895128255283200 to 7332895128243560448. Current Snapshot: MvccSnapshot[applied={T|T < 7332895128255283200}]
I20260924 13:46:39.680452 9752 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } } }
I20260924 13:46:39.680523 9752 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7 [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:39.680830 9723 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc4f8d231364aa59e1e72b30f9a59a7" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 39473 } } peers { permanent_uuid: "905fd32ffae84079a39ccde4b3c168ff" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 34105 } } peers { permanent_uuid: "a81b45a849a14dde8e00aecae2249bcf" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 41769 } } }
I20260924 13:46:39.680905 9723 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:46:39.686451 9755 catalog_manager.cc:1347] Generated new cluster ID: ac67135f357645c1b87257fc8d62b357
I20260924 13:46:39.686514 9755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:39.693732 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.678412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.19.53.253:34105)
0924 13:46:39.678446 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.678448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.686049 (+ 7601us) server_negotiation.cc:407] Connection header received
0924 13:46:39.686150 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.686151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.686161 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.686190 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.687106 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.687857 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.688895 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.689179 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.689217 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.690831 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.690832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.690833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.690840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.692458 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.692885 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.692886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.692887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.692950 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.693171 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.693172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.693173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.693294 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.693363 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.693561 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.693643 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:46:39.701519 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.678203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.19.53.254:39473)
0924 13:46:39.678239 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.678241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.694573 (+ 16332us) server_negotiation.cc:407] Connection header received
0924 13:46:39.694676 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.694677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.694688 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.694716 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.695384 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.696099 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.697080 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.697368 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.697406 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.698715 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.698716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.698717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.698723 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.700288 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.700693 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.700694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.700695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.700759 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.700948 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.700949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.700949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.701061 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.701140 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.701360 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.701431 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 13:46:39.706358 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:39.678719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.19.53.252:41769)
0924 13:46:39.678765 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.678767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.698157 (+ 19390us) server_negotiation.cc:407] Connection header received
0924 13:46:39.698260 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.698261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.698271 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.698298 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.698876 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.699698 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.701852 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:39.702156 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:39.702206 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:39.703662 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:39.703664 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:39.703665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:39.703673 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:39.705218 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.705558 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.705558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.705559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.705604 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:39.705775 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:39.705775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:39.705776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:39.705886 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:39.705983 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:39.706176 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 13:46:39.706254 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:46:39.718736 9755 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:39.719465 9755 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:39.726495 9755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acc4f8d231364aa59e1e72b30f9a59a7: Generated new TSK 0
I20260924 13:46:39.726725 9755 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:46:39.748435 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:39.746784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.19.53.254:39473)
0924 13:46:39.746828 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.746831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.747770 (+ 939us) server_negotiation.cc:407] Connection header received
0924 13:46:39.747877 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.747878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.747891 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.747920 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.748364 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46980: BlockingRecv error: recv got EOF from 127.0.0.1:46980 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 13:46:39.749344 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:39.747018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.19.53.253:34105)
0924 13:46:39.747059 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.747061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.748819 (+ 1758us) server_negotiation.cc:407] Connection header received
0924 13:46:39.748909 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.748911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.748920 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.748948 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.749289 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51574: BlockingRecv error: recv got EOF from 127.0.0.1:51574 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 13:46:39.750202 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:39.747494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43944 (local address 127.19.53.252:41769)
0924 13:46:39.747531 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.747533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.749692 (+ 2159us) server_negotiation.cc:407] Connection header received
0924 13:46:39.749785 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.749786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.749796 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.749823 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.750134 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43944: BlockingRecv error: recv got EOF from 127.0.0.1:43944 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 13:46:39.752738 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:39.751979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.19.53.253:34105)
0924 13:46:39.752014 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.752015 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.752228 (+ 213us) server_negotiation.cc:407] Connection header received
0924 13:46:39.752332 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.752333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.752350 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.752380 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.752698 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51590: BlockingRecv error: recv got EOF from 127.0.0.1:51590 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 13:46:39.755863 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:39.755109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.19.53.253:34105)
0924 13:46:39.755142 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.755144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.755391 (+ 247us) server_negotiation.cc:407] Connection header received
0924 13:46:39.755486 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.755487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.755497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.755525 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.755823 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51604: BlockingRecv error: recv got EOF from 127.0.0.1:51604 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 13:46:39.759881 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:39.759070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.19.53.253:34105)
0924 13:46:39.759103 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.759104 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.759387 (+ 283us) server_negotiation.cc:407] Connection header received
0924 13:46:39.759482 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.759483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.759493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.759521 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.759840 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51620: BlockingRecv error: recv got EOF from 127.0.0.1:51620 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 13:46:39.765861 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:39.765063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.19.53.253:34105)
0924 13:46:39.765101 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.765102 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.765360 (+ 258us) server_negotiation.cc:407] Connection header received
0924 13:46:39.765452 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.765453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.765462 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.765490 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.765815 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51632: BlockingRecv error: recv got EOF from 127.0.0.1:51632 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 13:46:39.775894 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:39.775013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.19.53.253:34105)
0924 13:46:39.775052 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.775054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.775330 (+ 276us) server_negotiation.cc:407] Connection header received
0924 13:46:39.775444 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.775446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.775459 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.775494 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.775848 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51638: BlockingRecv error: recv got EOF from 127.0.0.1:51638 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 13:46:39.794289 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:39.793328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.19.53.253:34105)
0924 13:46:39.793369 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.793371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.793677 (+ 306us) server_negotiation.cc:407] Connection header received
0924 13:46:39.793780 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.793781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.793793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.793823 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.794227 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51650: BlockingRecv error: recv got EOF from 127.0.0.1:51650 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 13:46:39.828490 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:39.827727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.19.53.253:34105)
0924 13:46:39.827804 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.827806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.827923 (+ 117us) server_negotiation.cc:407] Connection header received
0924 13:46:39.828000 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.828001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.828014 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.828089 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.828423 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51660: BlockingRecv error: recv got EOF from 127.0.0.1:51660 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 13:46:39.894383 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:39.893650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.19.53.253:34105)
0924 13:46:39.893742 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:39.893744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:39.893790 (+ 46us) server_negotiation.cc:407] Connection header received
0924 13:46:39.893867 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:39.893868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:39.893878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:39.893980 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:39.894325 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51674: BlockingRecv error: recv got EOF from 127.0.0.1:51674 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 13:46:40.024497 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:40.023766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.19.53.253:34105)
0924 13:46:40.023867 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:40.023869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:40.023932 (+ 63us) server_negotiation.cc:407] Connection header received
0924 13:46:40.023999 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:40.024000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:40.024010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:40.024063 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:40.024443 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51686: BlockingRecv error: recv got EOF from 127.0.0.1:51686 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 13:46:40.282188 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:40.281644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.19.53.253:34105)
0924 13:46:40.281712 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:40.281714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:40.281722 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:46:40.281779 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:40.281779 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:40.281789 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:40.281836 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:40.282142 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51700: BlockingRecv error: recv got EOF from 127.0.0.1:51700 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 13:46:40.619364 9720 catalog_manager.cc:1259] Loaded cluster ID: ac67135f357645c1b87257fc8d62b357
I20260924 13:46:40.619449 9720 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff: loading cluster ID for follower catalog manager: success
I20260924 13:46:40.620231 9720 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff: acquiring CA information for follower catalog manager: success
I20260924 13:46:40.620529 9720 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 905fd32ffae84079a39ccde4b3c168ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:46:40.638504 9734 catalog_manager.cc:1259] Loaded cluster ID: ac67135f357645c1b87257fc8d62b357
I20260924 13:46:40.638599 9734 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf: loading cluster ID for follower catalog manager: success
I20260924 13:46:40.639379 9734 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf: acquiring CA information for follower catalog manager: success
I20260924 13:46:40.639674 9734 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a81b45a849a14dde8e00aecae2249bcf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:46:40.797971 9834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:40.795487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.19.53.253:34105)
0924 13:46:40.795585 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:40.795587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:40.795598 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:46:40.795645 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:40.795646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:40.795701 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:40.795786 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:40.795969 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:40.796477 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:40.797236 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:40.797421 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:40.797470 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:40.797612 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:40.797733 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:40.797790 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:40.797820 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 13:46:40.797838 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":51,"threads_started":1}
I20260924 13:46:40.802009 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:40.799847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.19.53.252:41769)
0924 13:46:40.799938 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:40.799940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:40.799948 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:46:40.800002 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:40.800002 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:40.800036 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:40.800101 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:40.800183 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:40.800625 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:40.801307 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:40.801490 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:40.801536 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:40.801689 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:40.801787 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:40.801838 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:40.801883 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 13:46:40.801899 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":44,"threads_started":1}
I20260924 13:46:40.806748 9834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:40.803802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.19.53.253:34105)
0924 13:46:40.803845 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:40.803847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:40.803941 (+ 94us) server_negotiation.cc:407] Connection header received
0924 13:46:40.804060 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:40.804061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:40.804072 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:40.804138 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:40.804263 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:40.804907 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:40.805880 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:40.806190 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:40.806253 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:40.806397 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:40.806426 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:40.806470 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:40.806653 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 13:46:40.806678 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:46:40.807081 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:40.803769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.19.53.254:39473)
0924 13:46:40.803912 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:40.803915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:40.803924 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:46:40.804007 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:40.804008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:40.804062 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:40.804151 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:40.804346 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:40.805124 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:40.806102 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:40.806400 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:40.806472 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:40.806651 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:40.806750 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:40.806816 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:40.806972 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 13:46:40.807003 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260924 13:46:40.807231 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:40.804018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.19.53.252:41769)
0924 13:46:40.804051 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:40.804053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:40.804357 (+ 304us) server_negotiation.cc:407] Connection header received
0924 13:46:40.804454 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:40.804456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:40.804466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:40.804519 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:40.804678 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:40.805316 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:40.806170 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:40.806437 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:40.806492 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:40.806686 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:40.806724 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:40.806752 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:40.807139 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 13:46:40.807164 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:46:40.808929 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 9484
I20260924 13:46:40.814378 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 9559
I20260924 13:46:40.819453 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 9634
2026-09-24T13:46:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2786 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:46:40 dist-test-slave-tfm9 krb5kdc[9851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:46:40 dist-test-slave-tfm9 krb5kdc[9851](info): set up 2 sockets
Sep 24 13:46:40 dist-test-slave-tfm9 krb5kdc[9851](info): commencing operation
krb5kdc: starting...
W20260924 13:46:41.839984 19671 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/test-user.keytab.
Sep 24 13:46:41 dist-test-slave-tfm9 krb5kdc[9851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:46:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:46:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
Entry for principal kudu/127.19.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab.
I20260924 13:46:41.957237 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:37983
--webserver_interface=127.19.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:37983,127.19.53.253:33831,127.19.53.252:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:46:42.035609 9864 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:46:42.035847 9864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:42.035877 9864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:42.035930 9864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:42.037045 9864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:42.037112 9864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:42.037137 9864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:42.037164 9864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:42.037190 9864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:42.039042 9864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:37983,127.19.53.253:33831,127.19.53.252:44179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.254.keytab
--principal=kudu/127.19.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.254:37983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:42.039379 9864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:42.039629 9864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:42.044512 9872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:42.044512 9869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:42.044606 9864 server_base.cc:1034] running on GCE node
W20260924 13:46:42.044486 9870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:42.044909 9864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:42.045179 9864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:42.046366 9864 hybrid_clock.cc:648] HybridClock initialized: now 1790257602046353 us; error 30 us; skew 500 ppm
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257602, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:42.048902 9864 init.cc:377] Logged in from keytab as kudu/127.19.53.254@KRBTEST.COM (short username kudu)
I20260924 13:46:42.049276 9864 webserver.cc:460] Webserver started at http://127.19.53.254:33271/ using document root <none> and password file <none>
I20260924 13:46:42.049466 9864 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:42.049506 9864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:42.049590 9864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:42.050635 9864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/data/instance:
uuid: "e310c38cd8854977a2058fa1ded76e2f"
format_stamp: "Formatted at 2026-09-24 13:46:42 on dist-test-slave-tfm9"
server_key: "4c6d9b03646b039ade6f8064e567982a"
server_key_iv: "6047523e017082c29102b390a98f6f31"
server_key_version: "encryptionkey@0"
I20260924 13:46:42.050932 9864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/wal/instance:
uuid: "e310c38cd8854977a2058fa1ded76e2f"
format_stamp: "Formatted at 2026-09-24 13:46:42 on dist-test-slave-tfm9"
server_key: "4c6d9b03646b039ade6f8064e567982a"
server_key_iv: "6047523e017082c29102b390a98f6f31"
server_key_version: "encryptionkey@0"
I20260924 13:46:42.052152 9864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:46:42.052830 9879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:42.053016 9864 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:42.053099 9864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/wal
uuid: "e310c38cd8854977a2058fa1ded76e2f"
format_stamp: "Formatted at 2026-09-24 13:46:42 on dist-test-slave-tfm9"
server_key: "4c6d9b03646b039ade6f8064e567982a"
server_key_iv: "6047523e017082c29102b390a98f6f31"
server_key_version: "encryptionkey@0"
I20260924 13:46:42.053153 9864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:42.075798 9864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:42.078171 9864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:42.078290 9864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:42.082103 9864 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.254:37983
I20260924 13:46:42.082159 9931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.254:37983 every 8 connection(s)
I20260924 13:46:42.082518 9864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/data/info.pb
I20260924 13:46:42.083504 9932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:42.085016 9932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257602, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:42.091810 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 9864
I20260924 13:46:42.091925 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-0/wal/instance
I20260924 13:46:42.092199 19671 external_mini_cluster.cc:1428] Setting key 6647b1294e4129b0f445aa4ecf4db200
I20260924 13:46:42.094388 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.085698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:37983 (local address 127.0.0.1:34180)
0924 13:46:42.085943 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.085983 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:42.086425 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:42.086667 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:42.086670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:42.086942 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:42.087100 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.087104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.088055 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.088055 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.088728 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.088730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.089173 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.089174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.089227 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.089661 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:42.089669 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:42.091078 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:42.092791 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.092798 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.092800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.093070 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.093290 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.093291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.093291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.093411 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.093890 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:42.093896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:42.094019 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 13:46:42.094095 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 13:46:42.094219 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":124,"threads_started":1}
I20260924 13:46:42.094460 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.085713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34180 (local address 127.19.53.254:37983)
0924 13:46:42.085869 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.085879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.086073 (+ 194us) server_negotiation.cc:407] Connection header received
0924 13:46:42.086540 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.086542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.086557 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.086637 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.087178 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.087972 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.088796 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.089087 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.089163 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.091152 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.091159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.091159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.091165 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.092712 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.093154 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.093156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.093156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.093229 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.093472 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.093473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.093473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.093799 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.093919 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.094160 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.094303 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260924 13:46:42.095139 9932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:46:42.095499 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.095372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:33831 (local address 127.0.0.1:52340)
0924 13:46:42.095419 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.095468 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 13:46:42.095650 9880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.53.253:33831: connect: Connection refused (error 111)
I20260924 13:46:42.096086 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.095970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:33831 (local address 127.0.0.1:52354)
0924 13:46:42.096026 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.096056 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 13:46:42.096235 9932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.253:33831: Network error: Client connection negotiation failed: client connection to 127.19.53.253:33831: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.19.53.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.253.keytab.
Entry for principal kudu/127.19.53.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.253.keytab.
I20260924 13:46:42.116087 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.253:33831
--webserver_interface=127.19.53.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.253.keytab
--principal=kudu/127.19.53.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:37983,127.19.53.253:33831,127.19.53.252:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:46:42.145510 9932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } attempt: 1
I20260924 13:46:42.146118 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.145966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:33831 (local address 127.0.0.1:52366)
0924 13:46:42.146028 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.146077 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 13:46:42.146628 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.146518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:33831 (local address 127.0.0.1:52380)
0924 13:46:42.146569 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.146598 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 13:46:42.146759 9932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.253:33831: Network error: Client connection negotiation failed: client connection to 127.19.53.253:33831: connect: Connection refused (error 111)
W20260924 13:46:42.194989 9939 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:46:42.195183 9939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:42.195199 9939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:42.195218 9939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:42.196213 9939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:42.196245 9939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:42.196259 9939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:42.196271 9939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:42.196283 9939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:42.197662 9939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:37983,127.19.53.253:33831,127.19.53.252:44179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.253.keytab
--principal=kudu/127.19.53.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.253:33831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.53.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:42.197839 9939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:42.198028 9939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:42.202668 9945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:42.202711 9947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:42.202765 9944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:42.202910 9939 server_base.cc:1034] running on GCE node
I20260924 13:46:42.203254 9939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:42.203523 9939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:42.204717 9939 hybrid_clock.cc:648] HybridClock initialized: now 1790257602204704 us; error 28 us; skew 500 ppm
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257602, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:42.207326 9939 init.cc:377] Logged in from keytab as kudu/127.19.53.253@KRBTEST.COM (short username kudu)
I20260924 13:46:42.207719 9939 webserver.cc:460] Webserver started at http://127.19.53.253:38271/ using document root <none> and password file <none>
I20260924 13:46:42.207913 9939 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:42.207953 9939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:42.208036 9939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:42.208988 9939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/data/instance:
uuid: "c93070b64c1b43c984d945406a882690"
format_stamp: "Formatted at 2026-09-24 13:46:42 on dist-test-slave-tfm9"
server_key: "b7c71e122702b3b8e7e1ac84ffa1f4f0"
server_key_iv: "0d9e8386a307b3d9d12ae8f6fda338de"
server_key_version: "encryptionkey@0"
I20260924 13:46:42.209280 9939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/wal/instance:
uuid: "c93070b64c1b43c984d945406a882690"
format_stamp: "Formatted at 2026-09-24 13:46:42 on dist-test-slave-tfm9"
server_key: "b7c71e122702b3b8e7e1ac84ffa1f4f0"
server_key_iv: "0d9e8386a307b3d9d12ae8f6fda338de"
server_key_version: "encryptionkey@0"
I20260924 13:46:42.210574 9939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:46:42.211278 9954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:42.211454 9939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:46:42.211522 9939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/wal
uuid: "c93070b64c1b43c984d945406a882690"
format_stamp: "Formatted at 2026-09-24 13:46:42 on dist-test-slave-tfm9"
server_key: "b7c71e122702b3b8e7e1ac84ffa1f4f0"
server_key_iv: "0d9e8386a307b3d9d12ae8f6fda338de"
server_key_version: "encryptionkey@0"
I20260924 13:46:42.211571 9939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:42.214887 9932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } attempt: 2
I20260924 13:46:42.215245 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.215128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:33831 (local address 127.0.0.1:52392)
0924 13:46:42.215183 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.215219 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 13:46:42.215600 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.215513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:33831 (local address 127.0.0.1:52408)
0924 13:46:42.215553 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.215583 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.253:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 13:46:42.215699 9932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.253:33831: Network error: Client connection negotiation failed: client connection to 127.19.53.253:33831: connect: Connection refused (error 111)
I20260924 13:46:42.239848 9939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:42.242332 9939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:42.242491 9939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:42.246197 9939 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.253:33831
I20260924 13:46:42.246248 10006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.253:33831 every 8 connection(s)
I20260924 13:46:42.246611 9939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/data/info.pb
I20260924 13:46:42.247587 10007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:42.249078 10007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:46:42.250674 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 9939
I20260924 13:46:42.250808 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-1/wal/instance
I20260924 13:46:42.251005 19671 external_mini_cluster.cc:1428] Setting key 9ded34380d289992cdcb86aed58bdeda
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257602, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.253@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.53.252@KRBTEST.COM; defaulting to no policy
I20260924 13:46:42.258276 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.249734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.19.53.254:37983)
0924 13:46:42.249796 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.249798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.250094 (+ 296us) server_negotiation.cc:407] Connection header received
0924 13:46:42.250606 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.250607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.250617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.250684 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.251496 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.251967 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.253061 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.253266 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.253348 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.255402 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.255404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.255404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.255412 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.256848 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.257266 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.257267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.257268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.257327 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.257570 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.257571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.257571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.257693 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.257768 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.258098 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.258173 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 13:46:42.258419 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.249735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:37983 (local address 127.0.0.1:34182)
0924 13:46:42.249960 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.250009 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:42.250488 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:42.250794 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:42.250799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:42.251134 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:42.251402 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.251408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.252059 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.252061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.252961 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.252963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.253367 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.253368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.253489 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.253975 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:42.253983 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:42.255292 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:42.256957 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.256964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.256966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.257176 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.257394 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.257395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.257395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.257502 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.257761 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:42.257767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:42.257948 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 13:46:42.258079 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 13:46:42.258215 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260924 13:46:42.258955 10007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.53.252@KRBTEST.COM" created.
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257602, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.253@KRBTEST.COM for kudu/127.19.53.253@KRBTEST.COM
I20260924 13:46:42.266407 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.259187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:33831 (local address 127.0.0.1:52418)
0924 13:46:42.259264 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.259275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:42.259317 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:42.259574 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:42.259575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:42.259686 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:42.259731 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.259732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.260497 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.260498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.261112 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.261115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.261613 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.261614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.261694 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.262128 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:42.262134 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:42.263705 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:42.265381 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.265382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.265382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.265539 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.265750 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.265750 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.265751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.265801 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.266110 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:42.266111 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:42.266146 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:46:42.266217 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 13:46:42.266305 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 13:46:42.266474 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.259281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.19.53.253:33831)
0924 13:46:42.259428 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.259433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.259450 (+ 17us) server_negotiation.cc:407] Connection header received
0924 13:46:42.259470 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.259472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.259486 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.259552 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.259793 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.260434 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.261171 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.261537 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.261591 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.263799 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.263806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.263807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.263813 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.265282 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.265627 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.265630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.265630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.265687 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.265869 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.265871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.265871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.266043 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.266131 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.266243 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.266355 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260924 13:46:42.267160 10007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:46:42.267468 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.267373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44179 (local address 127.0.0.1:40908)
0924 13:46:42.267410 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.267449 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 13:46:42.267621 9958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
I20260924 13:46:42.268019 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.267947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44179 (local address 127.0.0.1:40922)
0924 13:46:42.267971 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.268005 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 13:46:42.268134 10007 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:44179: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.53.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.252.keytab.
Entry for principal kudu/127.19.53.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.252.keytab.
I20260924 13:46:42.276089 19671 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrxQH3m/build/release/bin/kudu
/tmp/dist-test-taskrxQH3m/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.252:44179
--webserver_interface=127.19.53.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.252.keytab
--principal=kudu/127.19.53.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.53.212:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.53.254:37983,127.19.53.253:33831,127.19.53.252:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:46:42.306846 9932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } attempt: 3
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257602, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for kudu/127.19.53.253@KRBTEST.COM
I20260924 13:46:42.313855 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.307199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:33831 (local address 127.0.0.1:52426)
0924 13:46:42.307264 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.307283 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:42.307340 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:42.307500 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:42.307501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:42.307665 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:42.307732 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.307734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.308408 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.308409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.308930 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.308931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.309322 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.309322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.309404 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.309891 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:42.309895 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:42.311239 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:42.312798 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.312799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.312799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.312969 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.313183 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.313183 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.313184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.313249 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.313516 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:42.313517 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:42.313563 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 13:46:42.313653 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 13:46:42.313758 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 13:46:42.313855 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.307307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.19.53.253:33831)
0924 13:46:42.307375 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.307377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.307388 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:46:42.307415 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.307416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.307432 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.307495 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.307837 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.308328 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.309016 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.309233 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.309321 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.311339 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.311344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.311345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.311352 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.312701 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.313038 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.313039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.313040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.313113 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.313329 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.313330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.313331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.313447 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.313525 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.313677 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.313758 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:46:42.314311 9932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:46:42.314705 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.314594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44179 (local address 127.0.0.1:40926)
0924 13:46:42.314645 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.314678 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 13:46:42.315004 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.314909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44179 (local address 127.0.0.1:40942)
0924 13:46:42.314953 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.314978 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 13:46:42.315078 9932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:44179: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
I20260924 13:46:42.317354 10007 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } attempt: 1
I20260924 13:46:42.317672 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.317570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44179 (local address 127.0.0.1:40948)
0924 13:46:42.317617 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.317650 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 13:46:42.317998 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.317874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44179 (local address 127.0.0.1:40958)
0924 13:46:42.317947 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.317971 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 13:46:42.318079 10007 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:44179: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
I20260924 13:46:42.346231 9932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } attempt: 1
I20260924 13:46:42.346698 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.346562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44179 (local address 127.0.0.1:40966)
0924 13:46:42.346638 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.346669 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 13:46:42.347044 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.346949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44179 (local address 127.0.0.1:40968)
0924 13:46:42.346993 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.347020 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 13:46:42.347126 9932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:44179: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
W20260924 13:46:42.357134 10014 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:46:42.357317 10014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:46:42.357357 10014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:46:42.357396 10014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:46:42.358543 10014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:46:42.358580 10014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:46:42.358594 10014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:46:42.358639 10014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:46:42.358660 10014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:46:42.360113 10014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.53.212:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.53.254:37983,127.19.53.253:33831,127.19.53.252:44179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/krb5kdc/kudu_127.19.53.252.keytab
--principal=kudu/127.19.53.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.53.252:44179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.53.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:46:42.360328 10014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:46:42.360528 10014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:46:42.365130 10019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:42.365240 10014 server_base.cc:1034] running on GCE node
W20260924 13:46:42.365152 10022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:46:42.365130 10020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:46:42.365564 10014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:46:42.365784 10014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:46:42.366915 10014 hybrid_clock.cc:648] HybridClock initialized: now 1790257602366902 us; error 24 us; skew 500 ppm
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257602, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:46:42.369359 10014 init.cc:377] Logged in from keytab as kudu/127.19.53.252@KRBTEST.COM (short username kudu)
I20260924 13:46:42.369796 10014 webserver.cc:460] Webserver started at http://127.19.53.252:40597/ using document root <none> and password file <none>
I20260924 13:46:42.370070 10014 fs_manager.cc:359] Metadata directory not provided
I20260924 13:46:42.370116 10014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:46:42.370258 10014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:46:42.371259 10014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/data/instance:
uuid: "14f646d5628946d8afe01303975ee888"
format_stamp: "Formatted at 2026-09-24 13:46:42 on dist-test-slave-tfm9"
server_key: "9f73398d5dfb2582b8c0bc32ef8c7fd0"
server_key_iv: "51450f3311726477e7631a8a86d5bab0"
server_key_version: "encryptionkey@0"
I20260924 13:46:42.371603 10014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/wal/instance:
uuid: "14f646d5628946d8afe01303975ee888"
format_stamp: "Formatted at 2026-09-24 13:46:42 on dist-test-slave-tfm9"
server_key: "9f73398d5dfb2582b8c0bc32ef8c7fd0"
server_key_iv: "51450f3311726477e7631a8a86d5bab0"
server_key_version: "encryptionkey@0"
I20260924 13:46:42.372864 10014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:46:42.373557 10029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:46:42.373725 10014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:46:42.373821 10014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/data,/tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/wal
uuid: "14f646d5628946d8afe01303975ee888"
format_stamp: "Formatted at 2026-09-24 13:46:42 on dist-test-slave-tfm9"
server_key: "9f73398d5dfb2582b8c0bc32ef8c7fd0"
server_key_iv: "51450f3311726477e7631a8a86d5bab0"
server_key_version: "encryptionkey@0"
I20260924 13:46:42.373903 10014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:46:42.381458 10014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:46:42.383569 10014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:46:42.383708 10014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:46:42.386188 10007 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } attempt: 2
I20260924 13:46:42.386601 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.386475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44179 (local address 127.0.0.1:40976)
0924 13:46:42.386530 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.386571 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 13:46:42.386958 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.386894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44179 (local address 127.0.0.1:40990)
0924 13:46:42.386919 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.386942 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 13:46:42.387064 10007 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.53.252:44179: Network error: Client connection negotiation failed: client connection to 127.19.53.252:44179: connect: Connection refused (error 111)
I20260924 13:46:42.387836 10014 rpc_server.cc:307] RPC server started. Bound to: 127.19.53.252:44179
I20260924 13:46:42.387864 10081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.53.252:44179 every 8 connection(s)
I20260924 13:46:42.388170 10014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/data/info.pb
I20260924 13:46:42.389081 10082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:46:42.390301 19671 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrxQH3m/build/release/bin/kudu as pid 10014
I20260924 13:46:42.390379 19671 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrxQH3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257335418480-19671-0/minicluster-data/master-2/wal/instance
I20260924 13:46:42.390568 19671 external_mini_cluster.cc:1428] Setting key b55913a777d10fa892ea9618c5a655fa
I20260924 13:46:42.390795 10082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257602, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.252@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.252@KRBTEST.COM
I20260924 13:46:42.399678 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.391496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34192 (local address 127.19.53.254:37983)
0924 13:46:42.391540 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.391542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.391662 (+ 120us) server_negotiation.cc:407] Connection header received
0924 13:46:42.391845 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.391846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.391871 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.391936 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.392441 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.392951 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.394078 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.394383 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.394458 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.396562 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.396563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.396564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.396570 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.398150 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.398695 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.398696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.398696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.398748 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.398997 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.398997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.398998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.399124 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.399191 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.399517 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.399594 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:46:42.399852 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.391409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.254:37983 (local address 127.0.0.1:34192)
0924 13:46:42.391550 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.391589 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:42.391762 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:42.391951 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:42.391954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:42.392186 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:42.392332 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.392336 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.393022 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.393023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.394009 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.394011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.394499 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.394500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.394627 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.395101 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:42.395109 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:42.396478 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:42.398224 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.398230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.398232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.398623 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.398813 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.398814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.398814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.398932 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.399244 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:42.399250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:42.399379 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 13:46:42.399484 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 13:46:42.399624 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":71,"thread_start_us":80,"threads_started":1}
I20260924 13:46:42.400382 10082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:46:42.400549 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.391011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.19.53.252:44179)
0924 13:46:42.391197 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.391207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.391233 (+ 26us) server_negotiation.cc:407] Connection header received
0924 13:46:42.391763 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.391770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.391792 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.391893 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.392576 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.393562 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.394515 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.394927 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.395021 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.397350 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.397356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.397356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.397361 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.399128 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.399453 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.399456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.399456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.399513 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.399703 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.399704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.399705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.399878 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.400345 (+ 467us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.400387 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.400460 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257602, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.252@KRBTEST.COM for kudu/127.19.53.253@KRBTEST.COM
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.254@KRBTEST.COM
I20260924 13:46:42.406960 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.400716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52432 (local address 127.19.53.253:33831)
0924 13:46:42.400767 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.400768 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.400919 (+ 151us) server_negotiation.cc:407] Connection header received
0924 13:46:42.400986 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.400987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.400994 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.401066 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.401342 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.401786 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.402487 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.402719 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.402790 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.404439 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.404440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.404441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.404447 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.405818 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.406176 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.406178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.406178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.406236 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.406443 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.406444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.406444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.406570 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.406644 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.406793 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.406872 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 13:46:42.406960 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.400643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.253:33831 (local address 127.0.0.1:52432)
0924 13:46:42.400856 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.400865 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:42.400931 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:42.401070 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:42.401071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:42.401196 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:42.401249 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.401251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.401874 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.401874 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.402366 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.402367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.402808 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.402809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.402875 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.403263 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:42.403267 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:42.404337 (+ 1070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:42.405904 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.405905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.405905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.406103 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.406302 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.406303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.406304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.406359 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.406641 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:42.406641 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:42.406681 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 13:46:42.406766 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 13:46:42.406871 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260924 13:46:42.407637 10082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:46:42.408769 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.401325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34202 (local address 127.19.53.254:37983)
0924 13:46:42.401368 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.401370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.401379 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:46:42.401398 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.401399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.401407 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.401458 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.401929 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.402357 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.403071 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.403278 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.403352 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.405868 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.405870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.405870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.405877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.407525 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.407932 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.407933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.407933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.407991 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.408216 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.408216 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.408217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.408343 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.408498 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.408533 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.408603 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257602, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.252@KRBTEST.COM for kudu/127.19.53.252@KRBTEST.COM
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.53.253@KRBTEST.COM
I20260924 13:46:42.414548 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.407940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41006 (local address 127.19.53.252:44179)
0924 13:46:42.407991 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.407993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.408001 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:46:42.408024 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.408025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.408035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.408084 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.408329 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.408993 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.409803 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.410074 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.410145 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.411785 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.411786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.411786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.411791 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.413260 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.413649 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.413650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.413650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.413730 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.413989 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.413990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.413990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.414121 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.414198 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.414339 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.414445 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":32}
I20260924 13:46:42.414738 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.407913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44179 (local address 127.0.0.1:41006)
0924 13:46:42.407949 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.407958 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:42.407989 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:42.408086 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:42.408087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:42.408227 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:42.408255 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.408256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.409067 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.409068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.409718 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.409720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.410144 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.410145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.410185 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.410509 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:42.410512 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:42.411705 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:42.413348 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.413349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.413349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.413583 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.413825 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.413826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.413827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.413894 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.414194 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:42.414195 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:42.414227 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 13:46:42.414298 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 13:46:42.414396 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 13:46:42.416290 10082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888: Bootstrap starting.
I20260924 13:46:42.416930 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.409399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.19.53.253:33831)
0924 13:46:42.409438 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.409440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.409448 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:46:42.409468 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.409469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.409477 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.409528 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.409933 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.410603 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.411330 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.411613 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.411667 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.414271 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.414272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.414273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.414279 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.415738 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.416118 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.416119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.416120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.416196 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.416412 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.416413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.416413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.416523 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.416730 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.416766 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.416838 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:46:42.417280 10082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:42.417630 10082 log.cc:824] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:42.418385 19671 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:46:42.418531 10082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888: No bootstrap required, opened a new log
I20260924 13:46:42.419328 9906 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e310c38cd8854977a2058fa1ded76e2f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34202
I20260924 13:46:42.420009 10082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } }
I20260924 13:46:42.420178 10082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f646d5628946d8afe01303975ee888, State: Initialized, Role: FOLLOWER
I20260924 13:46:42.420284 10082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } }
I20260924 13:46:42.420598 10082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:42.420586 10088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } } }
I20260924 13:46:42.420697 10088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 13:46:42.422204 10099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:46:42.422282 10099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:46:42.422281 9932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } attempt: 2
I20260924 13:46:42.422705 10014 master_runner.cc:186] Error getting master registration for 127.19.53.254:37983: OK, instance_id { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" instance_seqno: 1790257602080972 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:46:42.425403 10014 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257602, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.254@KRBTEST.COM for kudu/127.19.53.252@KRBTEST.COM
I20260924 13:46:42.429315 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.422496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44179 (local address 127.0.0.1:41014)
0924 13:46:42.422551 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.422561 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:42.422595 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:42.422722 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:42.422723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:42.422895 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:42.422938 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.422940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.423770 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.423771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.424629 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.424632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.425101 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.425102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.425173 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.425576 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:42.425581 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:42.426766 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:42.428298 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.428299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.428300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.428472 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.428678 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.428679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.428679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.428741 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.429009 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:42.429010 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:42.429052 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 13:46:42.429135 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 13:46:42.429226 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 13:46:42.429315 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.422588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.19.53.252:44179)
0924 13:46:42.422622 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.422624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.422633 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:46:42.422651 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.422651 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.422659 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.422712 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.423012 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.423688 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.424722 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.425016 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.425084 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.426874 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.426875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.426876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.426882 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.428200 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.428552 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.428553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.428553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.428602 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.428803 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.428804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.428804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.428930 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.429002 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.429158 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.429226 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 13:46:42.430572 9932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f: Bootstrap starting.
I20260924 13:46:42.431183 9932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:42.431461 9932 log.cc:824] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:42.432190 9932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f: No bootstrap required, opened a new log
I20260924 13:46:42.433113 9932 raft_consensus.cc:348] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } }
I20260924 13:46:42.433209 9932 raft_consensus.cc:729] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e310c38cd8854977a2058fa1ded76e2f, State: Initialized, Role: FOLLOWER
I20260924 13:46:42.433288 9932 consensus_queue.cc:260] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } }
I20260924 13:46:42.433560 10102 sys_catalog.cc:455] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } } }
I20260924 13:46:42.433621 9932 sys_catalog.cc:564] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:42.433634 10102 sys_catalog.cc:458] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:46:42.433748 9906 raft_consensus.cc:482] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 13:46:42.433838 9906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:42.434615 9906 raft_consensus.cc:504] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } }
I20260924 13:46:42.434845 9906 leader_election.cc:288] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [CANDIDATE]: Term 1 election: Requested vote from peers c93070b64c1b43c984d945406a882690 (127.19.53.253:33831), 14f646d5628946d8afe01303975ee888 (127.19.53.252:44179)
I20260924 13:46:42.436232 10056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e310c38cd8854977a2058fa1ded76e2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "14f646d5628946d8afe01303975ee888"
I20260924 13:46:42.436364 10056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:42.437179 10056 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e310c38cd8854977a2058fa1ded76e2f in term 1.
I20260924 13:46:42.437395 9883 leader_election.cc:302] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f646d5628946d8afe01303975ee888, e310c38cd8854977a2058fa1ded76e2f; no voters:
W20260924 13:46:42.435680 10113 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:46:42.437502 10113 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:46:42.437546 10102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:46:42.437669 10102 raft_consensus.cc:686] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [term 1 LEADER]: Becoming Leader. State: Replica: e310c38cd8854977a2058fa1ded76e2f, State: Running, Role: LEADER
I20260924 13:46:42.437745 10102 consensus_queue.cc:237] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } }
I20260924 13:46:42.438215 10121 sys_catalog.cc:455] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e310c38cd8854977a2058fa1ded76e2f. Latest consensus state: current_term: 1 leader_uuid: "e310c38cd8854977a2058fa1ded76e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } } }
I20260924 13:46:42.438364 10121 sys_catalog.cc:458] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:42.439087 9864 master_runner.cc:186] Error getting master registration for 127.19.53.253:33831: OK, instance_id { permanent_uuid: "c93070b64c1b43c984d945406a882690" instance_seqno: 1790257602245162 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:46:42.441355 9864 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:46:42.444886 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.437347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.19.53.252:44179)
0924 13:46:42.437384 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.437386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.437394 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:46:42.437412 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.437413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.437421 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.437474 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.438175 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.438878 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.439921 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.440206 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.440247 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.441902 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.441903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.441903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.441933 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.443585 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.443978 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.443979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.443979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.444036 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.444251 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.444252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.444253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.444397 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.444469 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.444695 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.444768 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:46:42.448287 10124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:46:42.448599 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.436281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.19.53.254:37983)
0924 13:46:42.436316 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.436317 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.436680 (+ 363us) server_negotiation.cc:407] Connection header received
0924 13:46:42.436781 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.436782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.436792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.436821 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.441798 (+ 4977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.442536 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.443555 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.443844 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.443882 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.445723 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.445724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.445725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.445732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.447323 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.447705 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.447706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.447707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.447760 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.447952 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.447952 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.447953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.448073 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.448149 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.448361 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.448437 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:46:42.448520 10124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:46:42.453038 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.437757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.19.53.253:33831)
0924 13:46:42.437814 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.437816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.440661 (+ 2845us) server_negotiation.cc:407] Connection header received
0924 13:46:42.440778 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.440779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.440789 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.440819 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.445853 (+ 5034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.446598 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.447574 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.447870 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.447914 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.449989 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.449990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.449991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.449998 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.451462 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.451807 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.451808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.451808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.451858 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.452035 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.452036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.452037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.452148 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.452664 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.452864 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.452941 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:46:42.455009 10056 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888 [term 1 FOLLOWER]: Refusing update from remote peer e310c38cd8854977a2058fa1ded76e2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:42.455330 10121 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:42.456691 10088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e310c38cd8854977a2058fa1ded76e2f. Latest consensus state: current_term: 1 leader_uuid: "e310c38cd8854977a2058fa1ded76e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } } }
I20260924 13:46:42.456790 10088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:46:42.457507 10102 sys_catalog.cc:455] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e310c38cd8854977a2058fa1ded76e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } } }
I20260924 13:46:42.457592 10102 sys_catalog.cc:458] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:42.457625 10088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e310c38cd8854977a2058fa1ded76e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } } }
I20260924 13:46:42.457717 10088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:46:42.457787 10126 mvcc.cc:205] Tried to move back new op lower bound from 7332895139652739072 to 7332895139585748992. Current Snapshot: MvccSnapshot[applied={T|T < 7332895139652739072}]
I20260924 13:46:42.457964 10125 mvcc.cc:205] Tried to move back new op lower bound from 7332895139652739072 to 7332895139585748992. Current Snapshot: MvccSnapshot[applied={T|T < 7332895139652739072}]
I20260924 13:46:42.466328 10124 catalog_manager.cc:1347] Generated new cluster ID: 80047ef9b8ad40459ac3ed53baa0f750
I20260924 13:46:42.466385 10124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:46:42.478183 10007 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } attempt: 3
Sep 24 13:46:42 dist-test-slave-tfm9 krb5kdc[9851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257602, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.53.253@KRBTEST.COM for kudu/127.19.53.252@KRBTEST.COM
I20260924 13:46:42.484192 10124 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:46:42.485172 10124 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:46:42.486555 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.478459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.53.252:44179 (local address 127.0.0.1:41032)
0924 13:46:42.478557 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 13:46:42.478571 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:46:42.478613 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:46:42.478807 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:46:42.478808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:46:42.479009 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:46:42.479060 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.479062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.479987 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.479988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.480805 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:46:42.480807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.481311 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.481312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:46:42.481368 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.481858 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:46:42.481863 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:46:42.483479 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:46:42.485301 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.485321 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.485321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.485538 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.485809 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:46:42.485810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:46:42.485811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:46:42.485881 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:46:42.486254 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:46:42.486255 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:46:42.486299 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 13:46:42.486362 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 13:46:42.486455 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 13:46:42.487775 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:42.478549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.19.53.252:44179)
0924 13:46:42.478591 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.478593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.478603 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:46:42.478691 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.478692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.478713 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.478760 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.479142 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.479924 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.480899 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:42.481189 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:42.481257 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:42.483572 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:46:42.483573 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:46:42.483574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:46:42.483581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:46:42.485154 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.485656 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.485657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.485657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.485728 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:46:42.485978 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:46:42.485979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:46:42.485979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:46:42.486114 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:46:42.486171 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:42.487609 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0924 13:46:42.487685 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:46:42.489588 10007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690: Bootstrap starting.
I20260924 13:46:42.490331 10007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690: Neither blocks nor log segments found. Creating new log.
I20260924 13:46:42.490639 10007 log.cc:824] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690: Log is configured to *not* fsync() on all Append() calls
I20260924 13:46:42.491703 10007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690: No bootstrap required, opened a new log
I20260924 13:46:42.493752 10007 raft_consensus.cc:348] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } }
I20260924 13:46:42.493878 10007 raft_consensus.cc:729] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93070b64c1b43c984d945406a882690, State: Initialized, Role: FOLLOWER
I20260924 13:46:42.493975 10007 consensus_queue.cc:260] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } }
I20260924 13:46:42.494299 10007 sys_catalog.cc:564] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:46:42.495854 9980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:46:42.495569 9981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e310c38cd8854977a2058fa1ded76e2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c93070b64c1b43c984d945406a882690"
I20260924 13:46:42.496719 9980 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690 [term 1 FOLLOWER]: Refusing update from remote peer e310c38cd8854977a2058fa1ded76e2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:46:42.497016 10121 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:46:42.497458 9981 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e310c38cd8854977a2058fa1ded76e2f in term 1.
I20260924 13:46:42.498149 10131 sys_catalog.cc:455] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } } }
I20260924 13:46:42.498237 10131 sys_catalog.cc:458] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 13:46:42.498937 10142 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:46:42.498986 10142 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:46:42.499038 10124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f: Generated new TSK 0
I20260924 13:46:42.499179 10124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:46:42.500775 10146 sys_catalog.cc:455] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e310c38cd8854977a2058fa1ded76e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } } }
I20260924 13:46:42.500907 10131 sys_catalog.cc:455] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e310c38cd8854977a2058fa1ded76e2f. Latest consensus state: current_term: 1 leader_uuid: "e310c38cd8854977a2058fa1ded76e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } } }
I20260924 13:46:42.501024 10131 sys_catalog.cc:458] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:46:42.500923 10146 sys_catalog.cc:458] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:46:42.501673 10102 sys_catalog.cc:455] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e310c38cd8854977a2058fa1ded76e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e310c38cd8854977a2058fa1ded76e2f" member_type: VOTER last_known_addr { host: "127.19.53.254" port: 37983 } } peers { permanent_uuid: "c93070b64c1b43c984d945406a882690" member_type: VOTER last_known_addr { host: "127.19.53.253" port: 33831 } } peers { permanent_uuid: "14f646d5628946d8afe01303975ee888" member_type: VOTER last_known_addr { host: "127.19.53.252" port: 44179 } } }
I20260924 13:46:42.501754 10102 sys_catalog.cc:458] T 00000000000000000000000000000000 P e310c38cd8854977a2058fa1ded76e2f [sys.catalog]: This master's current role is: LEADER
I20260924 13:46:42.502779 10144 mvcc.cc:205] Tried to move back new op lower bound from 7332895139768737792 to 7332895139585748992. Current Snapshot: MvccSnapshot[applied={T|T < 7332895139652739072}]
W20260924 13:46:42.529498 9934 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:42.528193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34226 (local address 127.19.53.254:37983)
0924 13:46:42.528252 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.528255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.528852 (+ 597us) server_negotiation.cc:407] Connection header received
0924 13:46:42.528954 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.528955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.528967 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.528997 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.529431 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34226: BlockingRecv error: recv got EOF from 127.0.0.1:34226 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 13:46:42.530458 10085 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:42.528554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.19.53.252:44179)
0924 13:46:42.528589 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.528590 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.529842 (+ 1252us) server_negotiation.cc:407] Connection header received
0924 13:46:42.530036 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.530037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.530047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.530073 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.530404 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41042: BlockingRecv error: recv got EOF from 127.0.0.1:41042 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 13:46:42.531560 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:42.530887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.19.53.253:33831)
0924 13:46:42.530923 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.530925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.531055 (+ 130us) server_negotiation.cc:407] Connection header received
0924 13:46:42.531144 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.531145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.531155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.531183 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.531487 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52454: BlockingRecv error: recv got EOF from 127.0.0.1:52454 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 13:46:42.533788 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:42.533090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.19.53.253:33831)
0924 13:46:42.533123 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.533125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.533303 (+ 178us) server_negotiation.cc:407] Connection header received
0924 13:46:42.533400 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.533401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.533416 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.533444 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.533745 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52464: BlockingRecv error: recv got EOF from 127.0.0.1:52464 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 13:46:42.539101 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:42.538285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.19.53.253:33831)
0924 13:46:42.538317 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.538318 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.538598 (+ 280us) server_negotiation.cc:407] Connection header received
0924 13:46:42.538690 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.538691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.538700 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.538729 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.539059 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52466: BlockingRecv error: recv got EOF from 127.0.0.1:52466 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 13:46:42.543183 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:42.542404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.19.53.253:33831)
0924 13:46:42.542436 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.542437 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.542693 (+ 256us) server_negotiation.cc:407] Connection header received
0924 13:46:42.542789 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.542790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.542801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.542830 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.543141 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52478: BlockingRecv error: recv got EOF from 127.0.0.1:52478 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 13:46:42.549189 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:42.548386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.19.53.253:33831)
0924 13:46:42.548420 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.548421 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.548702 (+ 281us) server_negotiation.cc:407] Connection header received
0924 13:46:42.548796 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.548797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.548807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.548835 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.549144 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52480: BlockingRecv error: recv got EOF from 127.0.0.1:52480 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 13:46:42.559389 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:42.558538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.19.53.253:33831)
0924 13:46:42.558582 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.558584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.558833 (+ 249us) server_negotiation.cc:407] Connection header received
0924 13:46:42.558932 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.558933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.558943 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.558972 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.559326 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52494: BlockingRecv error: recv got EOF from 127.0.0.1:52494 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 13:46:42.577909 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:42.576973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.19.53.253:33831)
0924 13:46:42.577039 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.577041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.577314 (+ 273us) server_negotiation.cc:407] Connection header received
0924 13:46:42.577420 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.577421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.577433 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.577462 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.577851 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52506: BlockingRecv error: recv got EOF from 127.0.0.1:52506 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 13:46:42.612205 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:42.611431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.19.53.253:33831)
0924 13:46:42.611479 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.611481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.611624 (+ 143us) server_negotiation.cc:407] Connection header received
0924 13:46:42.611722 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.611722 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.611733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.611793 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.612132 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52516: BlockingRecv error: recv got EOF from 127.0.0.1:52516 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 13:46:42.678256 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:42.677461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.19.53.253:33831)
0924 13:46:42.677560 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.677562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.677637 (+ 75us) server_negotiation.cc:407] Connection header received
0924 13:46:42.677715 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.677716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.677726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.677779 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.678153 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52526: BlockingRecv error: recv got EOF from 127.0.0.1:52526 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 13:46:42.808331 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:42.807543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.19.53.253:33831)
0924 13:46:42.807610 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:42.807612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:42.807744 (+ 132us) server_negotiation.cc:407] Connection header received
0924 13:46:42.807811 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:42.807812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:42.807823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:42.807864 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:42.808242 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52532: BlockingRecv error: recv got EOF from 127.0.0.1:52532 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 13:46:43.066442 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:46:43.065631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.19.53.253:33831)
0924 13:46:43.065716 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:43.065718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:43.065732 (+ 14us) server_negotiation.cc:407] Connection header received
0924 13:46:43.065827 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:43.065828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:43.065838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:43.065890 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:46:43.066376 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52538: BlockingRecv error: recv got EOF from 127.0.0.1:52538 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 13:46:43.422716 10099 catalog_manager.cc:1259] Loaded cluster ID: 80047ef9b8ad40459ac3ed53baa0f750
I20260924 13:46:43.422785 10099 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888: loading cluster ID for follower catalog manager: success
I20260924 13:46:43.423547 10099 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888: acquiring CA information for follower catalog manager: success
I20260924 13:46:43.423803 10099 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14f646d5628946d8afe01303975ee888: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:46:43.499447 10142 catalog_manager.cc:1259] Loaded cluster ID: 80047ef9b8ad40459ac3ed53baa0f750
I20260924 13:46:43.499553 10142 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690: loading cluster ID for follower catalog manager: success
I20260924 13:46:43.500450 10142 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690: acquiring CA information for follower catalog manager: success
I20260924 13:46:43.500751 10142 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c93070b64c1b43c984d945406a882690: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:46:43.582125 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:43.579593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.19.53.253:33831)
0924 13:46:43.579734 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:43.579736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:43.579745 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:46:43.579831 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:43.579832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:43.579881 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:43.579963 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:43.580145 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:43.580626 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:43.581361 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:43.581543 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:43.581592 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:43.581750 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:43.581858 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:43.581953 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:43.581971 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 13:46:43.581996 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260924 13:46:43.585819 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:43.583815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.19.53.252:44179)
0924 13:46:43.583910 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:43.583912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:43.583921 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:46:43.583970 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:43.583970 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:43.584002 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:43.584068 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:43.584141 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:43.584574 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:43.585173 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:43.585348 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:43.585393 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:43.585521 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:43.585599 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:43.585645 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:43.585696 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 13:46:43.585711 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":50,"threads_started":1}
I20260924 13:46:43.590261 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:43.587456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.19.53.253:33831)
0924 13:46:43.587493 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:43.587495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:43.587748 (+ 253us) server_negotiation.cc:407] Connection header received
0924 13:46:43.587867 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:43.587868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:43.587879 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:43.587932 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:43.588081 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:43.588559 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:43.589536 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:43.589758 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:43.589819 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:43.589984 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:43.590014 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:43.590065 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:43.590159 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 13:46:43.590183 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:46:43.590725 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:43.587474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.19.53.254:37983)
0924 13:46:43.587596 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:43.587598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:43.587671 (+ 73us) server_negotiation.cc:407] Connection header received
0924 13:46:43.587755 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:43.587756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:43.587809 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:43.587885 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:43.588061 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:43.588840 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:43.589729 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:43.590061 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:43.590116 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:43.590295 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:43.590400 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:43.590451 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:43.590611 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 13:46:43.590633 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260924 13:46:43.591486 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:46:43.587762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.19.53.252:44179)
0924 13:46:43.587801 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:46:43.587803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:46:43.587809 (+ 6us) server_negotiation.cc:407] Connection header received
0924 13:46:43.587909 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:46:43.587911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:46:43.587921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:46:43.587970 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:46:43.588147 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:43.588851 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:43.590682 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:46:43.590974 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:46:43.591015 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:46:43.591202 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:46:43.591234 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:46:43.591291 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 13:46:43.591380 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 13:46:43.591397 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:46:43.601755 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 9864
I20260924 13:46:43.607386 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 9939
I20260924 13:46:43.612287 19671 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrxQH3m/build/release/bin/kudu with pid 10014
2026-09-24T13:46:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2792 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5578 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (268200 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST